OpenMarket – November 29, 2019
I have learned a thing or two this year and returned with it to share with my office friends. Here’s how I entered the world of machine learning with a fancy toy car and used it as a learning resource at OpenMarket.
I am Tomasz Ptak, a boring Java dev by trade, passionate about de-legacy-fication – simplifying product development by making sure our technical debt doesn’t get stuck in the shutting door behind us. I am also racing in the AWS DeepRacer League by name Breadcentric. OpenMarket is serious about personal development and shaping a career path of their employees. In 2018 when I travelled to AWS re:Invent 2018, it was a learning roller-coaster to me, to say the least.
The AWS DeepRacer workshop opened a new world to me: write in code what you think matters in racing and see a simulated car go around the track using (and abusing) your logic.
Just to give a brief introduction: AWS DeepRacer is a way to put machine learning in engineers’ hands – literally. It does all the heavy lifting of:
- defining the problem (racing a car around the track)
- defining the training approach (selecting the algorithm, choosing the tools, defining the simulation of a car)
- setting up the training (setting up the machines, propagating the configuration)
- delivering results (transferring the model onto the car)
All you need to do to start is provide a reward function (with a choice of defaults to begin with), select a track and go.
At that time I understood the training process roughly this way:
The unicorn image has been released under Pixabay License on Pixabay
Jokes aside, DeepRacer is an example of a problem being solved using reinforcement learning – a type of machine learning in which the agent (car) makes decisions, initially mostly random tries (exploration), eventually trained ones (exploitation) and each of them is either rewarded or punished. The reward/punishment is used to calculate improvements to the car’s “decision process”.
Thanks to DeepRacer I could start applying machine learning within minutes and while the car was happily going around the track, I could focus on understand more aspects of it, like how to analyse the training, how to tune the training, then what the learning algorithm really does, how to alter the training and spin it up directly in SageMaker and Robomaker, and what problems arise from transferring solutions from simulation to real life. And the ultimate aspect: how to go faster than 1300 fellow racers, qualify into the finals of the AWS DeepRacer League and brag about it during every single coffee break.
AWS DeepRacer Workshop at the office
The ease with which one can start is amazing. I wanted to share it with my colleagues for their benefit as well. But first things first, I had to plan the key aspects.
What do I want to do it for?
- Show an interesting way to learn new things, and to help others learn them
- Strengthen collaborative learning
- Use me as an example that one doesn’t need to be a rock star to try something new and seize opportunities that it brings
- Build a group in the company that would be taking part in the races with me
OpenMarket London office is a combination of engineers and non-technical employees. While tech chats are usually attended by the technical staff, I tried to be prepared for people from sales or marketing. It is an interesting topic after all (just replace “machine learning” with “AI”). I especially hoped for members of the teams to which this subject is directly applicable to projects they are working on now or will be in the future.
I also had to take a couple things in to account:
- no prior knowledge – most of the colleagues only heard about DeepRacer from me so I had to plan a training simple enough to explain the concept with ease. A chat over lunch is not a training
- no prior teaching experience – I have talked at meetups and conferences, but it all has happened in the last month of my life (thanks to DeepRacer). Presenting and leading others is not an easy task for a training noob
- no place for a track – it is difficult to transfer the model from simulation into reality without a track that reflects the one in simulation with right colours and barriers; our offices are not tiny, but there is no space to put a 5 by 8 metre track without impacting fire safety
While interacting with the community I have seen reports of big events at other companies and organisations. While it would be nice to have something like that, I had to keep it realistic. I decided to start with a well-established format – a tech chat.
At OpenMarket we regularly organise tech chats – 30-60 minute presentation in which the speaker can share their interests and experiences, combined with questions and discussions. Tech chats are recorded and shared internally for all employees to benefit from. You get a chance to listen about Unicode, writing a programming language, why we should use Kotlin, what some of our legacy systems do and why and many, many more.
The good thing about tech chat is that it’s not just limited to technologies used at work and can lead to new goals. It also gives you quick feedback.
About fifteen people attended the tech chat where I talked about DeepRacer and my experience with it. While it is an awesome subject on its own, qualifying to the finals at AWS re:Invent 2019 does add an encouraging factor. I finished my talk with invitation for others to reach out so that we can race together.
I recommend recording your talks and watching them afterwards. You might have to get used to listening to your voice but it’s a valuable resource to show opportunities for improvement and to prove to yourself that it was good when you think it wasn’t.
After the chat I was asked to run a workshop where the volunteers could set up their own training. This time I led it as a whiteboard session which gives a good opportunity to explain the more practical aspects of training. Volunteers were asked to bring their laptops with AWS accounts and permissions to follow me as I set up a training.
I tried to stay within one hour and not to have a very strict plan for the session. This gave me the opportunity to practice my speaking skills and ability to carry out a training session.
Don’t be afraid of talking and don’t be afraid of not knowing the answers to questions. You don’t have to be an expert to show something interesting to others. Your greatest power is ability to say “I don’t know the answer, I will check and get back to you after the session”.
I emphasised that it is not mandatory to take part in the hands-on part of the workshop yourself, one can sit next to someone who does. This attracted similarly sized group of which six people started training.
DeepRacer encourages collective learning and making mistakes is critical element of it as is trying things out to see what happens and asking questions that make the speaker sweat.
My recommendation for the first workshop is to go with the defaults. The only three things you must do to start a first training is clicking the “Create resources” button, giving a name to your model about to be trained and clicking the start button. It takes about 10-15 minutes to give a thorough explanation of the console but it is better to skim over it without too many details. You don’t normally present matrices when students are learning to count to ten.
The only change I have made in the training was writing my own reward function. It promotes driving in London: roughly on the road, roughly on the left, not causing damage is desirable:
def reward_function(params): # Read input parameters all_wheels_on_track = params['all_wheels_on_track'] is_left_of_center = params['is_left_of_center'] reward = 1 if not is_left_of_center or not all_wheels_on_track: reward = 0.4 # Always return a float value return float(reward)
We started with rather low training speeds which meant that we could quickly see the results: my London driver was trained on the re:Invent 2018 track and was able to complete the re:Invent 2019 track after just two hours of training.
I didn’t say it would be fast though. Source: a screenshot from personal AWS DeepRacer console
Playing with the car
I have a car and I keep it at the office, but we have some limitations to how we can use it:
- First of all, fire safety (and general safety) is something we care about. We will not put it on risk even for the most awesome toy that I’ve ever had. Putting a 8x5m barrier is a no-no.
- Without barriers we could go for a printed carpet/vinyl, but it would still have to be put in the walking space which would not be practical.
- We could use a masking tape, but I did test it and while using advanced training to add some noise to the pictures helps the car deal with varying background details that the barriers isolate you from, we have a big issue on our floor: two roughly uniform colours of the carpet, joined in a straight line. My tests look like it reminds the track more than a tape and makes the car leave the road.
It was pure coincidence (I swear, I didn’t do it!) that the London meetup happened a day after the workshop. Even if people aren’t confident enough to see what their models can do, being able to interact with the community, learn from the experts and seeing others simply enjoy each others as they learn new things is a very motivating factor.
OMies at a DeepRacer 101 session during the London meetup. Used with permission from the author, David Smith, AWS
Plans going forward
Currently I am trying to nudge my colleagues regularly to make sure they try some more training. Even if you don’t have a chance to get a car and a track, you can focus on the virtual league. Currently (November 2019) there are two ongoing races happening online that give you a chance to challenge yourself and try to improve before the AWS DeepRacer League 2020.
And what a league it will be: 24 virtual races with winners taking part in the finals. There will also be more physical races around the globe. And stereo vision and LIDAR will introduce new types of challenges. Read the announcement.
Once I’m back from the finals I plan to set up weekly open tips and tricks sessions where we can discuss problems, present new solutions and techniques, and find ways to transfer this experience into real life problems at OpenMarket.
Ideas of subjects for such sessions include:
- log analysis to improve the outcomes of the training – including log analysis sessions
- maths behind Proximal Policy Optimisation
- training using AWS SageMaker
- trying to find the right place to set up a track at the office
Additionally, AWS have prepared private leagues and I plan to set up an in-house competition.
A couple tips and tricks
To finish of, I’d like to share a couple suggestions that may help you provide a better learning experience from AWS DeepRacer:
- If you haven’t done much DeepRacer yet, I recommend watching and reading AWS resources to help you prepare
- Determine goals, target group and constraints for your office. The format that worked for me might not be what you should replicate
- You might have the resources to put a bit more into it like inviting community experts for a talk or someone with a track and cars
- Look out for meetups, summits and other racing events in your area. They boost learning and entertaining. Also, an average DeepRacer racer is a very friendly and enthusiastic person that will help you solve problems and learn even more. If you don’t find a meetup, reach out to community to help you organise the first one
- NEVER EVER refer to DeepRacer as a toy for grown-up kids in front of your manager or HR. This is a very capable learning resource which leads to deeper understanding of concepts. The entertainment and challenge is there to boost the focus on learning. It leverages collective capability of the team. It stimulates team building and leads to even higher quality of deliverables
- I haven’t talked about who should pay for the training. It’s worth checking your options when it comes to which account to use
- If your company allows for using the company accounts, you need to make sure you get the right permissions. While there is a cost associated with it, first models will cost under $10 per user for the first two hours of training. This is enough to complete a lap in a race or two. After the stated training time all services shut down and S3 storage cost is marginal, that said you can reset all the resources and delete the buckets and you’re done
- The alternative would be to use personal free tier accounts and giving out some learning credits to support it. This way the cost side of DeepRacer is contained and your company accounts are focused around what matters the most
- Talk to your manager, HR and the marketing team – DeepRacer has a potential that can attract potential candidates, and prizes can be won. Some companies use it to build relations with customers as well. In some cases it may mean that putting “@CompanyName” in your racing name can get you support through a season of training. That will make it easier for you to learn
- When doing your performance review, don’t forget to list this twice: as a machine learning education, and as human learning. Chances are that this is the first time you are focusing not as much on improving your new technical skills as on building new soft skills with helping others learn as the main motivation
- Join the AWS DeepRacer Community at deepracing.io – we have the knowledge, experience and solutions to help you plan this and always do our best not to leave any questions unanswered.