Weekly Devlog – Jan 20th - 26th 2018

In contrast to last week, this week was abysmal, culminating in some unintended destructive testing. The demise of my Samsung S6 was quite unnerving and spectacular to watch.

Omelette, Snow & Godot

On Sunday after a miserably wet and then heavily snowy walk through Victoria Park to sample Mr Cyrus Todiwala’s excellent food at Park Café, in this case a fantastic Algerian omelette; the first ever omelette I have had that was genuinely uplifting I worked through a bit of the Godot 2 documentation and tutorials, continuing to be amazed by the quality of the engine and what it can do.

By the way, there’s an excellent recipe from Katie Loves Cooking here if you want to try it.

Cowbell

Monday started with a decision to abandon bringing the existing production code for Cowbell into line with the code base for my other apps as the more modern code has moved on to far and it would take too much effort to update. I decided to do a rewrite cloning off from my latest Travelling Santa Problem code base.

runtime

Once the initial version of my running app runtime is released, I’ll be following a roadmap of features which I think will make it appeal to a wide range of runners and non-runners alike. This roadmap will include an Open Street Map capability and to that end I’ve been building OSM VM servers and proving integration with osmDroid.

It’s a while since I’d worked on that area so a fair bit of time was spent updating the virtual machines. I’ll post something more on my planned initial infrastructure solution for that at a later date.

Whilst ensuring that map tiles were still served up properly to the Android client, I noticed that the map view would render correctly as it always has, then it would suddenly collapse down to a small section of the map. This area of the code has not changed for a long time so this behaviour was a puzzle.

I also noticed that the distance calculations were not updating and on one emulator the landing screen was failing to render completely. This was not going well. The TNG phrase “Bridge, we have a new problem” came to mind. It would render on the first launch after a fresh install, then nothing.

Clearing the head by taking a break

I am not fond of too many moving pieces on a project, so this was unsettling. I must admit that this day I really thought I should jack it all in. It was a tough day, but I was inspired by listening to “The Life Scientific” with Jim Al-Khalili and the fascinating Eugenia Cheng. Taking a break, listening to the programme whilst doing the dishes gave me the time to clear my head and dive back in.

The missing landing screen issue may have turned out to have been solved by following this advice from Stack Overflow. I handle all my screen navigation by modifying view visibility and switching from View.INVISIBLE to View.GONE seemed to do the trick. I’m hoping this change may also clear up the map view collapsing issue.

Playing nice with threads

I went for a short run in the afternoon to clear my head again and see how the app was behaving in the real world. This short test showed that the distance run view continued to show zero metres run (sometimes) and I noticed that my battery was was (sometimes) being drained at a rapid rate.

It turned out that I was not seeing any fresh points being returned from the database for the distance calculation to work on and I suspected this was a thread starvation issue.

I spotted that I was probably spanking the database and updating the internal data model too often, so the updates were backed off to be every five seconds and the various pieces could now get a chance to run.

Location, Location, Location

The next weird problem, which had me stumped for a while was around handling GPS permissions and the enabling of the device location service.

Previously in Android, use of the location permission could just be stated declaratively in the manifest, but now things are a bit tighter and the user needs to be asked to grant the permission.

Android also has an API which will check if granting of a permission also requires a rationale to be presented to the user explaining the reason for the permission.

All of this is laudable and should over time help clean up the Play Store of overly personal data hungry apps. As some of you will know I implement a Cookie Consent Popup in my apps which is a legal requirement for EU originated apps, but I have only ever seen one other app be compliant in this area.





Location permission handling is split up into two parts. The first is getting the user to grant access to the device location, this is where the rationale dialog comes in. The second is the enablement of the location service on the device itself. This is where my weird bug came in.

The weird bug was preceded by a classic schoolboy error as evidenced in the screenshot below of the code before and after. Apparently exclamation marks are important in Java! What an idiot!





The runtime app uses an unbound service to receive the device location from the network and GPS and then broadcast it on to the runtime Android client. I need to handle the case when the service is running and the particular location provider is disabled.


If I detect that the device’s location service is off, I display a dialog to the user and give them the option to go to the location settings. This is started in a separate activity, the user does their thing and then the user returns to the runtime app and goes about their business.



The problem was, the dialog was not being dismissed. It took me ages to figure out what was happening, but not before I had panicked and adorned the code with various guarding conditions, dismissed the dialog in different ways etc.

I even followed the trail of this red herring when I spotted an error on my Marshmallow emulator, but not any other.

01-25 06:26:11.674 E/Surface: getSlotFromBufferLocked: unknown buffer: 
01-25 06:26:11.676 onClick goToSettingsButton

It turns out there’s a bug in this version of Marshmallow (6.0.1). I burned a bit more time on that.

The penny dropped when I added this line of debug logging in.

01-25 08:01:40.188 D/runtime: ask user to explicitly enable location in settings 

01-25 08:01:40.548 D/runtime: ask user to explicitly enable location in settings 

01-25 08:01:40.557 D/runtime: ask user to explicitly enable location in settings

The dialog was dismissing perfectly…..only to reveal another instance of the dialog immediately behind. What a twit I am! AtomicBoolean guarding added so we only get one dialog instance and we were on our way again.

Wet, Wet, Wet

My next test was to check the accuracy of a run with an HTC One and S6 running runtime side by side. It was a damp day and the first part of the run went well.

Then, the heavens opened and there was no way I could keep the rain off both of the phones. I have a pretty good cover for the S6, the Spigen Tough Armor which has been an excellent case for the Galaxy and really helps with the grip as well. The S6 naked is like holding a wet mackerel in your hand, it just wants to get away and fall.

The water must have ingressed into the exposed USB port, headphones port and the volume down control as the screen started to be unresponsive, then started randomly launching apps, Samsung VR seemed to be the main one, giving me a rainbow coloured screen and frenetically switching between apps.

I could not get the phone shut down and got home as soon as possible to get it dried off. It could not have been more that a drop of two of rain that may have got inside, but it was enough.

Only afterwards did I find out that the S6 does not apparently have an IP rating as they wanted it to look sleek and cool more than they wanted it to be durable. This could be an expensive brick and one of my main development devices was now dead.

This was not a good week at all, wishing I was lucky.

Devlog Weekly Update – Jan 15th - 19th 2018

This week went reasonably well, working on Cowbell and runtime. Cowbell is my crowd support, shake to ring alpine bell app which was the first Android app I released back in September 2015.

Cowbell

As my app framework developed, the code for Cowbell started to fall behind in the metric I call build distance. Build distance can be used as an indicator of how out of date code is.

Cowbell was getting pretty out of date as my other projects progressed and its entropy increased. Also I’d started to notice the occasional crash in Cowbell and so I've started to devote more effort into bringing it into line with my other projects in respect of screen handling etc.

I also wanted to add capabilities which I’d been folding into my more recent apps such as alpha / beta tester detection and reward handling, promotion codes, rating and reviews and work on better monetisation.

These recent capabilities could not be simply folded into Cowbell as it stood. So began a lengthy and ongoing process of getting Cowbell to build cleanly and then be morphed and factored to use my latest code.

There is an imperative for this work in that the NFL Superbowl is coming up in February and the Virgin London marathon is coming up in April. Both of these could be good times for getting people to try Cowbell.

Cowbell had seen a nice spike on the day of the London marathon in the past when I'd pushed it and it would be nice to get the same exposure again if possible. I definitely need to work on retention though!

runtime

Work on runtime progressed towards its first closed alpha release to a small group on the 15^th January. The bug backlog was attacked with fervour.

Yesterday I received some strange looks as I tested the latest build out on Hackney Marshes. It is really quite difficult to run with two 7” tablets and two phones in your hands so I could compare results.

runtime will get another parkrun course test this morning, though it is horribly rainy this morning. It’s a good test for me as our local parkrun course features several loopbacks on itself which will exercise the GPS point clustering code I’ve written and is of course 5km long, so I can check its accuracy.

runtime allows the user to upload runs to Strava. I had been having trouble with the OAuth2 authorisation flow as the user may authenticate via three different means and they may do this on a device (in my case) in the API level range of 18 – 26. API changes meant that I thought I would have to handle multiple client types for Marshmallow, Nougat and “other”.

Luckily this week I have managed to get that down to one client type and barring some weird rendering behaviour on my old (API 18) HTC M1 test device, the flow appears to work for facebook, Google and email/password authorisation use cases.

I think (hope) I may have fixed an irritating defect which resulted in the distance run accuracy of runtime being off. runtime uses an unbound background service to handle position gathering and broadcasts that to the runtime client which handles persistence and all further processing.

It occurred to me that is was possible that when the user swaps out the client app to take a photo, answer a call etc. that position updates were being missed and when the app was resumed, the distance calculation and hence the projected time calculation was out.

I've now added a communication mechanism between the client app and the service so that it knows when it needs to queue up positions and broadcast them to the client when the client comes back. Hopefully this will fix my issue, I guess I’ll find out this morning.

Accuracy seems to have been improved. I'm reasonably happy with that for now.

Godot

As some light relief from working on bugs, I’d been browsing reddit and came across a post about the games engine Godot. I’m generally quite sceptical about these frameworks as I don’t like the ones that tie you into a particular style of game, but this looked different, interesting, flexible and free! I had been studying Unity and its support for Android VR.

Some of the showcased Godot apps are of excellent quality and I must admit, the engine has piqued my interest. I may consider a “quick” foray and roll one of my game ideas out in Godot to see if it will fly.

One of the showcased apps made with Godot which really caught my eye and I thought was excellently implemented, eye catching and really fun to play was Get Teddy.

I may use that as a reference benchmark for any work I do in Godot in the future.

Devlog Weekly Update – January 8th - 14th 2018

This was a frustrating week. Primarily I worked on getting runtime ready to go out to closed alpha test. The three main problem areas were location permissions, Strava authentication and AVD generation (always a bug bear, especially after a release of Android Studio).

The Android API level determines how location permissions are handled and it is no longer sufficient just to declare these in the Android manifest. I implemented additional dynamic checks on location permissions with an explanation to the user why these are needed and tested these changes on a number of emulators and physicals across a number of API levels.

This threw up a bug in the Oreo emulator which I initially thought was me. The system UI would crash whenever the user was shown the Allow/Deny location permission dialog.

It took me a while to track this down on StackOverflow and for now I’ll be avoiding testing runtime on Oreo, which is a risk, but I can’t really do anything until a fix is rolled out it seems.

For light relief I did a little bit more work on how I should be doing marketing during development and trying to get a feel for what I should be doing regarding initial launch.

On the Strava side, I have to do some jiggery pockery to get OAuth2 playing nice with Android within the runtime app and though the Email/Password and Google authorisation flows seem to work okay for runtime, facebook authorisation is not proven yet and needs to be completed.

This is not helped by the fact that I now have to support three authorisation flow mechanisms for Nougat, Marshmallow and “other” Android flavours. It is very frustrating.

On the upside (ish), I met up with some friends for beers and they dug out and played “Your word against mine” with each other, but, highlighting the great frustration that probably all independent developers have; the lack of feedback from users; they managed to find a multiplayer bug which can incorrectly decide the outcome of a game.

The lack of an un-install hook that would allow a user to leave some comment or feedback from Android users on the Play Store is very frustrating and it is depressing to have to wait until over a year after multiplayer was launched to find a bug like that and for it to be found in front of your eyes.

Due to the complexities of testing the multiplayer mode with Google Play Services on physical devices, I’ve not yet been able to recreate this bug to work on it. It’s high on the “to do” list!

Finally the week ended with more frustration over Android AVDs and the support for Android’s many screen formats. In contrast to the much simpler world of Apple devices, due to the open source nature of Android at the time of writing there are at least 15026 (17^th January 2018) different Android devices potentially to support (15001 a few days ago in the Google device catalogue), from a variety of manufacturers.

Android has 4 general screen sizes, any number of physical screen sizes, 14 different screen densities and currently API levels 10 through 27 potentially can be supported. The runtime app explicitly supports API levels 18 through 26 and I will probably have to do some additional filtration as behavioural results come in after launch.

The Google Play Analytics app gives a nice pretty overview of which devices users are using, but it is really of limited help. What the developer needs is the ability to either download or easily create an AVD which matches the specifications of the users device so we can test as closely as possible to the real thing. Sadly this is not available and I’m really not sure why it not straightforward for Google or someone to provide this. It is certainly a service I would be interested in if it could accurately generate an AVD.

Unfortunately you cannot generate an AVD which matches particular density, width, height and size characteristics. Android Studio lumps these into the standard densities (but not all of them) and certainly not all that the users are actually using. So I cannot match the user's device to an AVD and therefore I don’t know how my app is rendering on their device which is a risk. I’m flying blind.

Apparently there used to be a capability to specify the display density from the command line when using the emulator, but this feature has been deprecated for some reason. More frustration.

Next week will be better, right?

Using Spoon to test multiplayer apps

During a good evening out with some mates last night they started playing my multiplayer word game "Your word against mine".

I've been intending to refresh the game and improve the graphics, but my mates managed to spot a bug where it is possible for both players to be told they have won in a head to head game.

So my focus this Friday morning is to try and get to the bottom of the problem and I'll defer the graphics refresh until I've sorted the functionality issue.

Many months ago I took a look at using Spoon as a way to test on multiple devices with a signed APK which was a requirement for Google's real time multiplayer library.

This was my original post on my general devlog which will be dusted off today and I'll be reminding myself of the steps required to get this working again on a physical device.

http://development.celtic-intuition.com/real-time-multiplayer-spoon

Oreo - Apparent bug in emulator found

Lost a couple of hours this morning on this lovely little bug in the Oreo emulator build (onKeyguardOccludedChanged) when working with location services.

Thought it was me, but for once it was not!

This SO post helped - onKeyguardOccludedChanged NPE

runtime - Adding comprehensive location permission handling

Today I put together comprehensive handling of the location permissions for runtime.
Location permissions are handled differently dependent on the Android API level and it is not sufficient to declare the required permissions in the Android manifest.
So now runtime will do additional dynamic checks on location permissions, explaining the need for these permissions to the user in addition to the checks that were already in place for earlier API levels.

Started working on runtime again today

Now 2018 is underway and I managed to get Travelling Santa Problem out for Christmas, I wanted to go back and work on runtime, my upcoming app to improve my parkrun times.

It'll have a few more interesting features beyond being just a running app, but I want to get a first version out as soon as possible.

I ran into a few problems with threading and run measuring accuracy at the tail end of last year so I've gone back and redesigned a load of stuff to reduce the number of threads and overhead in calculating the distances.

Today went well with a slow 4K run on part of my local parkrun route. The app behaved nicely and my Strava track data agrees with my calculations.