2011 was a really intense year here at I.ndigo. More rewarding than launching 29 apps, was being able to witness the Brazilian market maturation and successfully accomplish worldwide recognized cases.

We would like to thank all partners, agencies, clients, employees and our families for believing in our potential and helping us build this result that we are pleased to share with you.

Bring on 2012!

Core Image was one of the many interesting topics discussed at iOS Tech Talk Tour that took place in São Paulo on january 9th. It is a framework that was already available at the MacOS and now can also be used by iOS developers.

It is important to notice that this framework is available only after iOS 5.0, resulting in a use limited to the application requirements. However, according to CNET the percentage of devices using iOS 5 in November, 2011 was already 40%, showing that apps developed to this version will shortly be available to the majority of users

Facial recognition is, by far, the most interesting of Core Image’s features, which will be detailed in this article. This new technique allow developers to think about new apps using this concept with a very low implementation cost.

We will show you how to implement the facial recognition straight from the device’s camera data stream. The source code is based on Apple’s SquareCam example project.

- (void)setupAVCapture
{
    AVCaptureSession *session = [AVCaptureSession new];
    if ([[UIDevice currentDevice] userInterfaceIdiom] == UIUserInterfaceIdiomPhone)
        [session setSessionPreset:AVCaptureSessionPreset640x480];
    else
        [session setSessionPreset:AVCaptureSessionPresetPhoto];
 
    // Select a video device, make an input
    AVCaptureDevice *device = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
    AVCaptureDeviceInput *deviceInput = [AVCaptureDeviceInput deviceInputWithDevice:device error:nil];
 
    if ( [session canAddInput:deviceInput] )
        [session addInput:deviceInput];
 
    // Make a video data output
    videoDataOutput = [[AVCaptureVideoDataOutput alloc] init];
 
    // we want BGRA, both CoreGraphics and OpenGL work well with 'BGRA'
    NSDictionary *rgbOutputSettings = [NSDictionary dictionaryWithObject:
    [NSNumber numberWithInt:kCMPixelFormat_32BGRA] forKey:(id)kCVPixelBufferPixelFormatTypeKey];
    [videoDataOutput setVideoSettings:rgbOutputSettings];
    [videoDataOutput setAlwaysDiscardsLateVideoFrames:YES]; // discard if the data output queue is blocked (as we process the still image)
 
    videoDataOutputQueue = dispatch_queue_create("VideoDataOutputQueue", NULL);
    [videoDataOutput setSampleBufferDelegate:self queue:videoDataOutputQueue];
 
    if ( [session canAddOutput:videoDataOutput] )
        [session addOutput:videoDataOutput];
 
    previewLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:session];
    [previewLayer setBackgroundColor:[[UIColor blackColor] CGColor]];
    [previewLayer setVideoGravity:AVLayerVideoGravityResizeAspect];
    CALayer *rootLayer = [previewView layer];
    [rootLayer setMasksToBounds:YES];
    [previewLayer setFrame:[rootLayer bounds]];
    [rootLayer addSublayer:previewLayer];
    [session startRunning];
 
}

Identifying a face with a CIDetector

According to CIDetector’s Class Reference the CIDetector object (available since iOS 5 inside CoreImage.framework) uses image processing to find “features” inside an image.

So, the next step is to identify the face in our video data stream is to configure a CIDetector. We can create an instance of the object by instantiating:

NSDictionary *detectorOptions = [[NSDictionary alloc] initWithObjectsAndKeys:CIDetectorAccuracyLow, CIDetectorAccuracy, nil];
 
faceDetector = [[CIDetector detectorOfType:CIDetectorTypeFace context:nil options:detectorOptions] retain];

When we previously initialized the camera, we configured our controller to act as a the video output stream’s delegate (videoDataOutput) at the line:

[videoDataOutput setSampleBufferDelegate:self queue:videoDataOutputQueue];

We can now implement the following method to read the video data stream:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
…
}

Finally, with a CIDetector’s instance and the video data stream, We can identify our face.

// got an image
CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CFDictionaryRef attachments = CMCopyDictionaryOfAttachments(kCFAllocatorDefault, sampleBuffer, kCMAttachmentMode_ShouldPropagate);
CIImage *ciImage = [[CIImage alloc] initWithCVPixelBuffer:pixelBuffer options:(NSDictionary *)attachments];
if (attachments)
    CFRelease(attachments);
NSDictionary *imageOptions = nil;
 
// '6' identifies device on vertical position
imageOptions = [NSDictionary dictionaryWithObject:[NSNumber numberWithInt:6] forKey:CIDetectorImageOrientation];
NSArray *features = [faceDetector featuresInImage:ciImage options:imageOptions];
[ciImage release];

The array features have each element as an instance of a CIFaceFeature, which identify a new face found in the video and allow us to retrieve several information from it.

Image’s CIFaceFeature

A CIFaceFeature object properties describes the face found in a image. These properties are:

hasLeftEyePosition
hasRightEyePosition
hasMouthPosition
leftEyePosition
rightEyePosition
mouthPosition

Besides that, due to its CIFeature inheritance, it also has the following properties:

bounds – The rectangle that the feature was found inside
type – The feature type

By using this information several actions can be taken, such as adding new visual elements on top of the face which was found in the image.

CIDetectorAccuracyLow Vs CIDetectorAccuracyHigh

When we created our CIDetector, we provided the CIDetectorAccuracyLow parameter:

NSDictionary *detectorOptions = [[NSDictionary alloc] initWithObjectsAndKeys:CIDetectorAccuracyLow, CIDetectorAccuracy, nil];

The reason we did that is because we are trying to read from a video stream, and using this option results in a faster analysis for each video frame, however, with a higher chance of not detecting any face at all.

In general, the CIDetectorAccuracyHigh option is used to analyse a single picture, resulting in a slower processing time, but with a higher face detection rate.

As you can notice, iOS 5 made it extremely easy to integrate facial recognition, which allows us to think again in several features that would be impracticable to implement in a project before. That said, we still have to be aware of the project requirements, since not all the users updates their operating system to the latest iOS version.

This post is also available in portuguese here.

As we defined before, this test will show the performance of 4 situations (see details on the previous post):

1. Insert without join tables;
2. Inserts with tables;
3. Select without join tables;
4. Select with join tables.

The previous post showed the first 2 situations. This time we will cover the last 2 (selects).

1. Select without join tables

This test tries to execute selects without joins in 2 ways:

• Fetch by object’s attributes;
• Fetch by identifier.

In each case, we see how the time to execute the select varies with the number of registries of the table (from 1 to 10000):

a) Fetch by object’s attributes

As we can see on the chart, when fetching by an attribute that is not indexed the time needed to execute the select varies almost linearly with the number of registries of the table. So, it is easy to think on how the performance of your table is getting worst with the time.

b) Fetch by identifier

This case shown us that fetching by the identifier (indexed), the time to fetch almost do not change with the number of registries, once the average time to fetch was almost equal the min time.

Conclusion

This tests resulted on the following table:

As we can see, for simple selects (without joins) when possible we should use identifiers to fetch, but, if we need to fetch by an attribute, it’s not hard to think about the performance, once it increases linearly with the size of the table.

2. Selects with join tables

This test shows how the time to execute a select increases as the number of join tables (in each select) and number of rows increases. The number of joins varies from 0 to 4.

a) Joins quantity: 0

This test has no joins, so the results is the same from the previous test when fetching by an attribute.

b) Joins quantity: 1

As might be expected, with 1 join the average time to a insert was much worst, about 4.27 times greater than with 0 joins.

c) Joins quantity: 2

With 2 joins we needed almost the same time to process the select, as expected, once the fetching engine has already entered on the process’ join step, what is not needed with 0 joins.

d) Joins quantity: 3

With 3 joins the average time was slightly worst again, as we might expect.

e) Joins quantity: 4

Again, with 4 joins the average time was slightly worst, as we might expect.

Conclusion

The following table results from the tests:

As we can see, we have a great variance from 0 to 1 join, but a small variance as the number of joins increases, due the way the select engine works.

This post, and the previous one, showed performance tests for Core Data that bring us information to analyse when use it in a project and what impact we would have when using it.

The next posts will present our analysis over the Magical Panda Active Record framework. We expect you are anxious as we are. Stay tuned!

This post is also available in Portuguese: Testes de performance do Core Data sobre SQLite – Parte 3.

According to Cocos2d website, Cocos2d for iPhone is a framework for building 2D games, demos, and other graphical/interactive applications. It is based on the cocos2d design: it uses the same concepts, but instead of using Python it uses Objective-C.

Their website says that Cocos2d for iPhone is:

• Easy to use: it uses a familiar API, and comes with lots of examples
• Fast: it uses the OpenGL ES best practices and optimized data structures
• Flexible: it is easy to extend, easy to integrate with 3rd party libraries
• Free: is open source, compatible both with closed and open source games
• Community supported: cocos2d has an active, big and friendly community (forum, IRC)
• AppStore approved: More than 550 AppStore games already use it, including many best seller games.

Cocos2d comes with an API that makes it simple to create an OpenGL ES based project, even if you are not an expert with OpenGL programming, once it has a nice encapsulation of some functionalities that are mostly used.

One negative point of using this kind of tool is that sometimes you get lost when it automates something that you didn’t want to be done for you. When trying to create a transparent OpenGL ES Layer with Cocos2d we saw that lot of people had this problem so we resolved to post about it.

Creating a Transparent OpenGL ES Layer with Cocos2d

Creating a transparent OpenGL ES Layer with Cocos2d v 0.99.4 was not an easy job, once its template code use a Macro to initialize a set of variables.

If you see the CC_DIRECTOR_INIT() macro, located on the ccMacros.h file, we have the following:

#define CC_DIRECTOR_INIT()                                                        \
do {                                                                              \
                                                                                  \
    window = [[UIWindow alloc] initWithFrame:[[UIScreen mainScreen] bounds]];     \
                                                                                  \
    if( ! [CCDirector setDirectorType:kCCDirectorTypeDisplayLink] )               \
        [CCDirector setDirectorType:kCCDirectorTypeNSTimer];                      \
                                                                                  \
     CCDirector *__director = [CCDirector sharedDirector];                        \
     [__director setDeviceOrientation:kCCDeviceOrientationPortrait];              \
     [__director setDisplayFPS:NO];                                               \
     [__director setAnimationInterval:1.0/60];                                    \
                                                                                  \
     EAGLView *__glView = [EAGLView viewWithFrame:[window bounds]                 \
                                      pixelFormat:kEAGLColorFormatRGB565          \
                                      depthFormat:0                               \
                               preserveBackbuffer:NO];                            \
                                                                                  \
     [__director setOpenGLView:__glView];                                         \
                                                                                  \
     [window addSubview:__glView];                                                \
     [window makeKeyAndVisible];                                                  \
                                                                                  \
 } while(0);                                                                      \

As we can see, this macro creates an EAGLView that has pixelFormat of type kEAGLColorFormatRGB565, which is 16 bits. In order to have transparency enabled in our EAGLView we need to create it using kEAGLColorFormatRGBA8 format, which is 32 bits.

We have lot of ways to solve it, such as changing that macro or initializing everything by ourselves. The important thing is to make sure to change the line:

EAGLView *__glView = [EAGLView viewWithFrame:[window bounds]                      \
                                 pixelFormat:kEAGLColorFormatRGB565               \
                                 depthFormat:0                                    \
                          preserveBackbuffer:NO];                                 \

to

EAGLView *__glView = [EAGLView viewWithFrame:[window bounds]                      \
                                 pixelFormat:kEAGLColorFormatRGBA8                \
                                 depthFormat:0                                    \
                          preserveBackbuffer:NO];                                 \

So, we will have a transparent layer when using:

glClearColor(0, 0, 0, 0);

In the above line, the last parameter indicates the opacity.
(http://www.khronos.org/opengles/sdk/1.1/docs/man/).

So that’s it! We hope you enjoy Cocos2d for iPhone and this tip helps you!

This post is also available in Portuguese: Cocos2d for iPhone 0.99.4 – Camada Transparente com OpenGL ES

As we defined before, this test will show the performance of 4 situations (see details on the previous post):

1. Insert without join tables;
2. Inserts with tables;
3. Select without join tables;
4. Select with join tables.

This post will cover the first 2 situations (inserts).

1. Inserts without join tables

This test tries to execute 10000 inserts on 5 different ways:

• Batch size: 1; Times: 10000;
• Batch size: 10; Times: 1000;
• Batch size: 100; Times: 100;
• Batch size: 1000; Times: 10;
• Batch size: 10000; Times: 1.

The results were the following:

a) Batch size: 1; Times: 10000

As we can see on the chart, we have a slight variance between the inserts. Although the maximum chart value was 0.57 seconds, the average (0.047s) was much more close to the minimum value (0.037s). Also, we can see that the time needed to a insert does not changes significantly while the table increases.

b) Batch size: 10; Times: 1000

This case shown us that, increasing the batch size to 10 we need, on average, about 1.64x more time to execute this batch. So, we can imagine that is much better to execute a big batch than lot of small batches. The test shown that to insert 10000 registries with batch size of 10 we needed 77.3 s, while with batch size of 1, to insert 10000 registries we needed 417.3 s.

c) Batch size: 100; Times: 100

This case shown that increasing again the batch size, now to 100, we needed about 3.6x more time per batch than we needed with batch size of 10. So, we can deduce that the time per batch does not increases linearly as the batch size increases. But again, the total time to execute 10000 inserts was lower (27 s against 77 s).

d) Batch size: 1000; Times: 10

Again, the same happened. As we increases the batch size to 1000 we needed 9.1x more time than we needed with batch size of 100. The total time was better than the previous, but is almost the same (25 s against 27 s).

e) Batch size: 10000; Times: 1 (10 repetitions with empty database)

This time, increasing the batch size to 10000 we needed 8x more time than we needed with batch size of 1000, while we could imagine we would need more than 9.1. So, we needed only 22 s to execute 10000 inserts.

Conclusion

This tests resulted on the following table:

As we can see, for simple inserts (without joins) as we increases the batch size, the time needed to to execute that batch is greater, but, the total time is lower. So, with Core Data, when possible we should save data to database using batches.

2. Inserts with join tables

This test shows how the time to execute a insert increases as the number of join tables (in each insert) and number of rows increases. The number of joins varies from 0 to 3.

a) Joins quantity: 0

This test has no joins, so the results is the same from the previous test.

b) Joins quantity: 1

As might be expected, with 1 join the average time to a insert was 1.17 times greater than with 0 joins.

c) Joins quantity: 2

With 2 joins we needed even more time to process an insert: 1.37 times more than with 1 join. Also, we can see that it seems that, as the database increases, we need more time to do inserts that was join tables.

d) Joins quantity: 3

With 3 joins the average time was worst again: 1.244 times more than with 2 joins.

Conclusion

The following table results from the tests:

As we can see, as the number of joins increases in a insert the time required to process it also increases. This increasing number is not linear.

The next post will present our results and conclusions for selects on Core Data, stay tuned!

This post is also available in Portuguese: Testes de performance do Core Data sobre SQLite – Parte 2

Introduction

Core Data for iPhone was introduced in the 3.0 version of the SDK, although it was previously available for Mac OSX. The framework implements an Object Graph Manager, giving applications the ability to manage data, including inserting new records, applying changes, undoing and redoing them and also the ability persist them.
It provides a high level API that abstracts all data management rules, as unique identifiers, model consistency and data validation, insertion, update and deletion, as well as data store creation and operation.

Data modeling abstraction is achieved by Xcode’s integrated graphical data modeling tool, where  the developer must describe an entity-relationship diagram representing and the system’s data. Xcode then generates all the classes and files needed to manage and optionally persist the data.

The persistent store layer abstracts the database and file store to the developer. The iPhone SDK provides SQLite and a binary format and the Mac OSX SDK also provides XML persistence. Either way, the implementation details stay apart from the developer, who should only care about objects and its properties.

Core Data Architecture

To implement the previously described functionality, a flexible and solid architecture was created, as seen in the following diagram:

Core Data Architecture

• NSManagedObjectModel: Created on run-time, based on the project’s data model, which is designed by the developer in the integrated graphical tool, represents the hole system’s data;

• NSManagedObject: Represents each entity and its properties, modeled by the developer. Those properties includes attributes and relationships;

• NSManagedObjectContext: Provides all the mentioned functionality to the Managed Objects, as fetching and deletion; It is aware of and has access to the Persistent Store Coordinator;

• NSPersistentStoreCoordinator: Provides an interface to the data persistence layer.

As mentioned before, Core Data is not limited to data persistence. All aspects of data management are separated from the persistence layer and can be used without it.

Testing Methodology

Environment

Data Model

Data Model

Testing Scenario

SQLite was chosen over binary format because it is the only type that is capable of partial object graph loading, making use of the lazy fetching feature, what is very important for the powerful, though limited iPhone hardware capabilities.

Also, a 10000 lines database, or 10000 persistent objects, was considered enough for performance testing purpose. Core Data persists its objects by saving all the NSManagedObjects present in the NSManagedObjectContext by once. Therefore, the testing methodology for the insertion tests included batch operations, with different of objects quantity per save, as the following table describes:

The following testes will be performed:

1. Insert without join tables
   The main objective of this test is to determine the insertion performance degradation, according to the database size. Therefore, following the previous ta ble, 10000 objects will be persisted.
2. Insert with join tables
   This test purpose is to decide how much the number of table joins affect the performance. Therefore, a smaller database will be used (2000 lines) with join quantity from none to four.
3. Select without join tables
   In this test two approaches will be considered: fetching objects by its attributes and getting them by their identifier. The performance is going to be measured, until the persisted object’s count reaches 10000.
4. Select with join tables
   This tests aims to identify the selection performance degradation, according to the number of object relationships (table joins). Since this is the heaviest test, once the data must be inserted and then selected accordingly, data will be selected after each 100 objects are inserted (atomically).

All tests have as main objective to determine the performance degradation of Core Data in relation to the amount of persisted data and the number of relationships between the data entities.

The next post will present our results and conclusions, stay tuned!

This post is also available in Portuguese: Testes de performance do Core Data sobre SQLite – Parte 1

We don’t know how the application will respond to these data access with the time (as the database size increases). Moreover, we have lot of third-party options of persistence layer, or layers responsible to manage data access and data mapping to objects.

These third-party options may have other problems that could resulting in performance loss with the time. So, choosing the best for each situation isn’t an easy job.

Thus, I.ndigo has decided to begin a series of performance experiments on the most known options of Persistent Store, and obviously, the Core Data purely implemented.

Searching on the web for options resulted on the following list of technologies: