Symfony Performance tips and tricks

Many a times, there is a knock on Symfony framework that it is slow. Frequently you will hear quips like “Symfony has a great set of features for building enterprise applications but performs worse than <insert your framework here>”. There are also many benchmarks which will take a trivial use case and an untuned Symfony and paint a sorry picture.

But we have rolled out pretty zippy applications that handle tons of loads using Symfony. We will share a few tips and tricks that we have used in our apps.

First off, performance is an important aspect and cannot be an afterthought. Often teams will finish the functionality and then do a ‘performance testing and fixing’ phase. It goes without saying that it is the wrong approach. The overall performance of the application is a culmination of many design decisions and implementation tricks and hence performance should be an exercise throughout the life cycle of the project.

Given that at Ideas2IT, we offer performance and scalability as a service offering itself and invested in a partnership with AppDynamics, we have developed a framework that follows the cycle of Estimate, Measure, Profile, Implement/Improve – Repeat, throughout the life cycle of a project to achieve desirable performance and scalability.

Estimate:
It is important to get an estimate of what is expected of the system in terms of

  • Load
  • Data Volume
  • Availability
  • Cost constraints

If you don’t do this exercise, we will under tune the system. And sometimes might over plan which can be a costly exercise.

Measure:
Never attempt to fix performance without measuring the performance profile of individual components at a detail level.

Use a tool to generate load with the characteristics of our estimate. Our tool of choice for this is JMeter. We have also used SAAS tools like Loadimpact to avoid the effort of setting up load generating hardware. It’s important that the load generator correlates to the estimated load. Don’t go too small or too high. We usually try 2-3X of expected load. One best practice is also to keep increasing the load to know the breaking points for future planning.

Profile:
Once you have your load suite running and you find bottlenecks in the system, please don’t try to fix it. Though it sounds common sense, often we have seen developers thinking of a clever idea and implementing it which might improve the performance of the component by 20% but not making any dent at all on the overall performance. Many times this is because the component’s contribution will be very small to the overall time taken.

Use profilers for each part of the application and spend enough time on profiling before attempting to fix. Our experience is that on a slow request typically 80-90% will go in a particular part of the request. May be a slow query or slow DOM binding for instance.

Our tools of choice for this:

  • FrontEnd – Google Chrome developer tools, PageSpeedInsightPhantomas
  • PHP layer – Blackfire, Symfony profiler
  • Database – Depends on the database. For instance, for MySQL, we use the inbuilt profiler and Jet profiler.

Implement/Improve:
There are a lot of strategies for improving each part of a complex application. Parts being frontend, Symfony / App layer and database.

FrontEnd:

Page speed is very important for user’s experience and often cited as the most important UX element these days. On top of it, if your application needs to be SEO friendly, then page speed is important as Google uses it as one of the variables to compute SEO rank.

There are a lot of things that can be done to improve page speed (independent of the application speed).
In any modern application, a large part of user’s perceived performance is because of the time taken to get the data/assets from server and rendering the UI. A typical profile of a request will look like

screenshot

Make sure your application gets an A on all metrics tracked by YSlow. Google maintains a nice set of rules for this:
https://developers.google.com/speed/docs/insights/rules.

Your biggest bang for the buck in terms of perceived performance will come from frontend tuning rather than Symfony/Appserver.

Database layer:

We will not go into details on what all we can do in the DB layer as it is a series of blogs in itself. But some pointers.

Slow query log:
Enable slow query log and make sure none of your queries do a full table scan and return within a threshold we have defined. Many times, teams will clear the slow query log and then forget about it. Then in one release, a developer will check in a nasty join with full table scan on a couple of big tables which will drag the whole application down. So set up a process as part of your QA to verify this on every release. If you are deploying an NPM tool like New Relic or AppDynamics, there are more sophisticated ways.

Many times, in spite of all right indices, you will find full table scans in slow query log. We will give one example, but there are many such cases.

# Time: 130816 11:40:12
# User@Host: root[root] @ localhost [127.0.0.1]
# Query_time: 7.168569  Lock_time: 0.000211 Rows_sent: 100000  Rows_examined: 200000
SET timestamp=1376633412;
select schedule0_.id as col_0_0_, 
schedule0_.event_name as col_1_0_,
schedule0_.description as col_2_0_,
schedule0_.location as col_3_0_, 
schedule0_.event_type as col_4_0_, 
schedule0_.start_date as col_5_0_, 
schedule0_.end_date as col_6_0_, 
schedule0_.start_time as col_7_0_,
schedule0_.end_time as col_8_0_,
schedule0_.remind_before as col_9_0_, 
schedule0_.remind_time_type as col_10_0_,
schedule0_.repeat_every as col_11_0_,
schedule0_.sunday as col_12_0_,
schedule0_.monday as col_13_0_,
schedule0_.tuesday as col_14_0_,
schedule0_.wednesday as col_15_0_,
schedule0_.thursday as col_16_0_,
schedule0_.friday as col_17_0_,
schedule0_.saturday as col_18_0_,
user2_.id as col_19_0_ from ETEC_SCHEDULE
schedule0_ inner join ETEC_EVENT_USERS
users1_ on schedule0_.id=users1_.event_id
inner join ETEC_USER user2_ on users1_.user_id=user2_.id where
(end_date between '2013-07-28' and '2013-09-08' or
schedule0_.end_date&gt;'2013-09-08') and user2_.id='admin';

 

In this instance it turned out to be because of the comparison to date in the where clause.

Just changing the where clause to fixed it.

schedule0_.start_date
&lt;='2013-09-08' AND schedule0_.end_date&gt;='2013-07-28'

 

Polygot persistence:
Start with a database that is a good fit for most of your use-cases. Often this is a good old RDBMS. Then for specific use-cases which have different characteristics, choose a different DB. For instance, in one of our social apps, the table that tracked cumulative user activity grew by millions per day. We moved this table alone to Cassandra.

In most of our applications, we deal with 2-3 databases.

Scale horizontally:
This is again a much discussed topic but there is only so much you can do to tune a database. From the beginning, follow a share nothing architecture to help you scale horizontally when the need arises.

Symfony/App layer:

Take care of the steps called out in Symfony documentation like byte code cache: http://symfony.com/doc/current/performance.html. One thing to note here that is different from what this documentation calls out is that the future of APC seems unstable. Consider using an alternative like Redis.

And do the basic things like upgrading PHP. Performance gains in higher versions of PHP are quite high. For instance, just the difference between PHP 5.4 and 5.3 is quite measurable.

Avoid n+1 query problem:
Often we will retrieve an object and an association to serve a page. One easy way of doing it is using Doctrine’s findAll method. For instance, to display a category and its products in an eCommerce site:

{% for category in products %}
<product>
<h2>{{ product.description }}</h2>
<p>Product: {{product.category.title }} {{ product. category.description }} created at {{ product.createdAt | date('d-m-Y H:i') }}</p>
</product>
{% endfor %}

 

But if you check the Symfony profiler, there will be one query for category and one query for each product in that category.

Instead, do an eager fetch as below.

public function findAllCategoriesAndProducts ()
{
$qb = $this->createQueryBuilder(‘p’);
$qb->addSelect(‘x’)
->innerJoin(‘c.product’, ‘x’);
return $qb->getQuery()->getResult();
}

 

Bulk updating entities:
This is a simple one, but novice developers to ORMs often do this to update an attribute in multiple entities:

$newExpiryAt = new \DateTime();
$product = $this->getDoctrine()->getRepository(‘AcmeDemoBundle:Product’)->findAll();
/** @var Product $product */
foreach ($products as $product) {
$product->setExpiryAt($newExpiryAt);
}
$this->getDoctrine()->getManager()->flush();

Instead bulk update using:

{
$qb = $this->createQueryBuilder(‘p’);
$qb->update()
->set(‘p.expiryAt’, ‘:newExpiryAt’)
->setParameter(‘newExpiryAt’, $newExpiryAt);
return $qb->getQuery()->execute();
}

Don’t bind to a container and then query for required service. Instead, inject the service directly. If you want to know all the available services in a large code base, you can do it by

Php app / console debug: container

 

Use a gateway cache to improve performance and bring down the load on app server.

Cache actions where you can:
In our experience, the biggest performance impact is often achieved by proper caching.
We already saw caching assets in browser cache, usage of CDN etc in the front end section.
In the app server layer, you can cache entire pages or parts of pages (fragments), and service level data. This drastically brings down the response time of the request. In addition, it brings down the overall load on the system so that even cache misses are served faster.

Caching pages or fragments:
Entire pages or part of pages can be cached with Vanish + ESI. Even in a dynamic application, there are views which is suitable for caching and need not be accurate with real-time information. For instance, comments on blog or recommendations. A good blog on caching fundamentals: http://2ndscale.com/rtomayko/2008/things-caches-do

Though Symfony has an inbuilt proxy, always use a purpose built reverse proxy like Varnish. Varnish consistently outperforms Symfony’s inbuilt proxy. https://www.symfony.fi/entry/symfony-benchmarks-symfony-proxy-vs-varnish

Caching service data:
This can be done either at the ORM level by caching query results etc or/and more higher level caches. Often times we end up using 2 or more cache layers – cache chaining pattern.

Few Symfony specific caching steps:

  • Enable query cache (this is different from query result cache). This converts the DQL query into its SQL counterpart. You would think it would be enabled by default, but its not.
  • Enable meta data cache. This caches the parse meta data from annotations or XML configurations instead of doing it on every request.
  • Cache query results for often requested queries on slow changing data. You can cache raw SQL result data or explicitly cache the hydrated object.
  • For really demanding applications, side step the ORM, directly get data as associative arrays using Doctrine DBAL and construct the object graph in the App layer.
  • Though declarative object graphs are nice from design perspective, they are performance night mares for ORMs. So use one-to-many and many-to-many associations only as needed. And almost always avoid bi-directional associations. If you do use a lot of associations, consider doing multi-step hydration.

How to setup Laravel5 on Azure

Laravel has become a leading PHP framework because of its many significant features. When deploying to Azure, there are a few nuances. Here we attempt to cover them.

Steps for Laravel on Azure:

  1. Install PHP version >= 5.4
  2. Install Composer
  3. Perform Directory Setup
  4. URL Rewriting

1. Install PHP version >= 5.4
Laravel5 requires PHP version >= 5.4. Azure provides an option to install the needed PHP version as per our project. To change the PHP version in Azure, follow the below steps

  • Log into Azure, and click the web service link. It will list all settings under your web service.
  • Go to `SETTINGS` and click `Application Settings` Link. It will list all app settings.
  • Go to `General Settings and choose the `PHP version` and save.

f1

2. Install Composer
Composer is an important package manager in PHP. It is mainly used with Laravel for package management. A composer can be easily installed on Azure via its extension manager.

  • Click your web service link. It will list all settings under your web service.
  • Go to `DEVELOPMENT TOOLS` and click on `Extensions`.
  • Click on the `add` link and select the option `Choose Extension`. It will list all Extensions related to your web service.
  • Select Composer from the list and do the installation process (you must agree to the license etc.).

f2

3. Perform Directory Setup
Directory setup is an important process in Laravel projects. Laravel lifecycle always starts from the file `public/index.php`. Azure deploys everything you push into the `wwwroot` directory.

To change the project start up point in Azure, follow the below steps:

  • Click your web service link. It will list all the settings under your web service.
  • Go to `SETTINGS` and click ` Application Settings` Link. It will list all app settings.
  • Go to `Virtual applications and directories and provide project folder path as (site\wwwroot\public)

f3

4. URL Rewriting
Generally, if we type an invalid URL on a laravel based site, we get redirected to 404 page.This occurs because of URL mod_rewrite rule.Laravel assumes that it is dealing with an Apache Web-Server, therefore it processes the mod_rewrite rules specific to that.But Azure Web Apps use Microsoft’s IIS.Hence, we need to include the below-mentioned changes for URL rewriting.
Create a new file called web.config in public directory with the below content.

<?xml version="1.0" encoding="utf-8"?>  
<configuration>  
  <system.webServer>
    <urlCompression doDynamicCompression="true" doStaticCompression="true" dynamicCompressionBeforeCache="true"/>
    <staticContent>
      <remove fileExtension=".svg" />
      <mimeMap fileExtension=".svg" mimeType="image/svg+xml" />
      <mimeMap fileExtension=".woff" mimeType="application/font-woff" />
      <clientCache httpExpires="Mon, 30 Mar 2020 00:00:00 GMT" cacheControlMode="UseExpires" />
    </staticContent>
<handlers>
      <remove name="OPTIONSVerbHandler" />
      <remove name="PHP54_via_FastCGI" />
      <add name="PHP54_via_FastCGI" path="*.php" verb="GET,HEAD,POST,OPTIONS" modules="FastCgiModule" scriptProcessor="D:\Program Files (x86)\PHP\v5.4\php-cgi.exe" resourceType="Either" />
    </handlers>
    <rewrite>
      <rules>
        <rule name="Laravel5" stopProcessing="true">
          <match url="^" ignoreCase="false" />
          <conditions logicalGrouping="MatchAll">
            <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" />
            <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
          </conditions>
          <action type="Rewrite" url="index.php" appendQueryString="true" />
        </rule>
      </rules>
    </rewrite>
  </system.webServer>
</configuration>

Wireframes and their importance in Mobile App Development

If you built a time machine 20 years ago and travelled to the present, one thing that would stick out is the ubiquity of the smartphone. From the premium iDevices to the budget smartphones available at one-tenth the price, there seems to be a phone made for everyone. It has evolved from being a status symbol to an absolute necessity to access the internet and the various services that run on it.

With a userbase that large, it is understandable that many companies would want their app to reserve a seat in their customer’s smartphone, and maybe occupy a special place in its notification drawer.

The advantage that companies like Ideas2IT have, is that we have been developing mobile apps from the very beginning, and have served such companies for a while now. From simple and elegant fitness apps to complex team communication and meeting apps, we have a wide range of development experience behind us. With this experience comes a certain level of understanding and a healthy amount of respect for the various steps involved in the process.

We can broadly divide the process of creating an app into three steps – gathering requirements (which may or may not be included in the brief), UI design, and then development. While each step is important, we noticed very early on that there is a hidden step, one that bridges the gap between the list of requirements on paper, to the polished UI. (Note: some readers might object to the omission of UX or User Experience. That is because UX design is intimately involved in every step of the process.)

Continue reading “Wireframes and their importance in Mobile App Development”

Online PAN Verification Process API Using PHP Laravel

 

Permanent Account Number (PAN) is a unique code that acts as an identification for Indian citizens who pay Income Tax. This 10-character alphanumeric identifier is assigned to all individuals identifiable under the Income Tax Act, 1961. An example is BJSPK7865D. This is issued by the Income Tax Authority of India and acts as a proof of identification. The primary usage of PAN is to identify and track all financial transactions and ensure the absence of tax evasion by tracking all economic transactions, especially those of high net worth individuals (HNI).

PAN has become an indispensable tool for KYC for India focused applications. Here is a short tutorial on how to integrate PAN verification into your app.

The PAN API Integration facility is an interface that allows the user to make an online verification of PAN by accessing the verification-site through a software application.

Following are the steps to achieve the verification process

  1. Digital Signature Certificate
  2. NSDL e-Gov Registration
  3. Server Setup
  4. Integration with Laravel

Continue reading “Online PAN Verification Process API Using PHP Laravel”

Is your Healthcare app HIPAA compliant?

HIPAA – stands for Health Insurance Portability and Accountability Act. HIPAA laws were enacted in 1996 years before the advent of iOS and Android devices. Smartphones have hitherto brought in a flood of apps in the Healthcare industry.

HIPAA aims at protecting the Health Information of patients’ medical records (PHI – Protected Health Information). Mobile apps will fall under HIPAA if the app deals with PHI, stores PHI or shares the PHI with healthcare providers (Doctors, dentists, pharmacies), Insurance companies, Government Programs, Health information processors.

 As a company that deals with Healthcare Domain and mobile app development, we have compiled a quick checklist of items to consider before you start the app development.

Continue reading “Is your Healthcare app HIPAA compliant?”

Aadhaar Verification in Laravel

A centralized Identity system a la SSN in the USA has been sorely lacking in India. But with the critical mass Aadhaar scheme has gained and the API ecosystem maturing around it, finally we have a viable option for identity.

We have been leveraging this in many applications to implement disruptive use cases like social lending.

Here, we are going to show how to integrate Aadhaar verification using Aadhaar Bridge  in Laravel.

Aadhaar Bridge Flow Diagram

Continue reading “Aadhaar Verification in Laravel”

Microservices – Service Discovery

In a monolithic architecture, we deployed a few application servers and statically registered them with load balancers. But in Microservices based architectures, inherently the services are mobile and can come and go because of auto-scaling, self-healing etc.

So the clients need a way for looking up the services they want to talk to. Essentially we need a way for mapping logical service names to one or more network paths.

 


 

Continue reading “Microservices – Service Discovery”

Android Things – Google latest addition to the world of IoT

As a mobile application development company that prides itself in leveraging the latest technology, the latest thing to catch our eye is Android things.

If you are a follower of google, you must have heard about “Brillo” – an operating system of google which has been now redesigned and branded as “Android Things”. It is a platform which uses Android software and helps us to make our most loved gadgets to talk to each other and allows us to utilize the cloud services to control them remotely.

Some insights on Android things,

1. A major highlight is the seamless integration of Android Things into Android Ecosystem.

2. Brillo was using C++ for development whereas Android things also targets java developers.

3. It has inbuilt support for Firebase which helps us to integrate it easily with Android Things.

4. Google has also added support for Weave, a standalone communication platform which allows devices to connect with google services.

5. Android Thing also has support to push updates which allow developers update the app seamlessly.

6. As of now not all google APIs for android are supported.

7. It already supports Intel’s Edison, Raspberry Pi 3 and NXP’s Pico out of the box which will give the developers a quick jump start to try building prototypes.

Agile Fixed Bid – An Oxymoron?

flexibility of agile
Ideas2IT is primarily a high-end IT development outsourcing firm. We have been used to take up projects on fixed bid for clients who have a definite set of requirements. Even then, there is almost always scope changes due to changing requirements or due to requirements being elaborated in a way which was not originally envisioned while estimating. This leads to us raising change requests and having a tough time convincing the client.

If this is so, how can we do a fixed bid contract for projects following the agile process?  Agile, by definition, has to be flexible to accommodate changes as the project progresses. And how would a fixed bid work? Isn’t it an oxymoron?

The Truth about Fixed Bids

If you have done any fixed bid outsourcing before you would have realized the following:

  • Fixed bid is best suited for projects which have their requirements well defined.
  • Short duration projects work well in fixed bid mode.
  • Any small change in scope entails a long process of estimation and approvals.
  • In longer duration projects, you are forced to make a lot of compromises as the project progresses with the intent of keeping the project on track.
  • Fixed bids mostly follow the water fall methodology making it rigid to accept any changes midstream.
  • Even when the iterative process is followed, the very nature of fixed bid doesn’t allow for much changes.

Continue reading “Agile Fixed Bid – An Oxymoron?”

An Introduction to Multipeer Connectivity in Mobiles

As a mobile application development company that takes pride in leveraging bleeding edge technologies to give our apps a business edge, Multipeer Connectivity is a technology we are excited about.

Now apps can send text messages, make calls and share files even without the internet! All this is made possible by the exciting new technology of Peer-to-Peer Mesh Network. Connectivity without cellular network is really powerful. This technology helps to create massive peer-to-peer mobile networks which opens up a whole world of possibilities for apps.

FIRECHAT is an iOS app that took the apps world by storm by introducing the concept of “Communicate without the Internet”. This became a lifesaver application for many during #ChennaiRains, #KashmirFloods and many other natural disasters all over the world. SERVALMESH is another example of how disruptive this technology is.

Continue reading “An Introduction to Multipeer Connectivity in Mobiles”