When I refer to current R&D status, I refer to the situation which was relevant for about 3 months ago. Not that today it is very different, but what started in eToro 3 months ago I’m going to call the future of eToro R&D. And I’m going to cover it in next post.

Anyway, lets talk about the current R&D status. We are big and we are getting bigger all the time. We are about 70 employees in R&D (including developers, QA, DBAs and IT).

We have migrated from SVN to Mercurial revision control system. SVN was really bad with dealing with branch merges. Openbook was a big team of 8 people and it was impossible to continue with SVN. We wanted to work using feature branch model. Our only two candidates were Git and Mercurial, both are very similar and fit our needs. We chose Mercurial because we are using FogBugz as our bug tracking system and FogCreek released Kiln, a Mercurial server which has a tight integration with FogBugz.

Shared static environments problems is a daily talk and everybody knows that dealing with configuration changes or fixing a broken environment can take hours, sometimes days. There is no integration environment where we can test cross R&D developments. We have to work in the office, because we are dependant in our network, we have no mocks.

We have a lot of dependencies, we wait for QA to finish, we wait for DBA to write the script, we wait for IT to deploy application, we wait for IT to see our logs in production in case of troubleshooting. We are doing blocking development.

We have sort of monitoring in production using LogEntries and New Relic which gives us more information, but we still can’t find the source of the problem when we have crisis.

In the past year we slowly started to learn and implement Test Driven Development process in few teams, writing unit tests. Still not enough, product can hardly understand/feel the impact of these tests. Same goes with peer code reviews. We have no test automation at all, our manual regression test takes 3 days of 2 QA people.

About a year ago we recruited configuration manager who started with build automation in teams. Nowadays most of the projects are built automatically. Deployments are still done manually to all environments.

We have no performance test lab, but we already starting to feel the impact of customers growth in production. All the problems are found in productions.

We have very few DBs, some are really big with a lot of shared objects, no APIs on shared objects. We have no scripts to create clean DB with just the schema. Our DBs should be separated to small DBs. DB revision control is awkward.

In the last year we made a transition in most of the teams from Waterfall to Agile software development methodology. It gave us a new breath, short releases (once in 2-3 weeks), more visibility of the process, tools to measure the process and get improved. From other hand it also emphasized our downsides, our dependencies, risks we take. It is still hard to manage it, we have delays in iterations, we commit to stories which we don’t deliver and we deliver stories which we didn’t commit to.

We feel that we are not scalable internally. We afraid to recruit another team member, because it won’t increase our performance, it will decrease it and I was able to prove that.

It is a heavy and slow machine which can hardly move…

P.S. Next post will be much more positive. ;)

It’s all started for me about 6 years ago, summer 2007. I left another startup in gambling industry in favor of eToro, which looked very promising to me. I had an experience of few years as webmaster and web developer and I was at my last year of computer science studies in a collage. My main mission was to establish company websites.

We were very few developers in a fresh new startup company. And as any typical Israeli startup company of these years our focus was to build and deliver our products as fast as possible.

There were about 5 developers all in all when each was responsible to cover products in multiple fields. As web developer I was responsible to build and maintain company websites, Affiliates system and Billing front-end.

In the first year the development process was almost unmanaged. We got requirements and requests for development by email or during a short brainstorm meeting in our CEO (Yoni Assia) office. We had no revision control system, neither bug tracking system. And it all worked fine, as soon as we were a small team.

About a year later, as our needs and requirements increased, we had to recruit more developers to the teams. R&D teams started to grow. We got more and more emails with development requests which we passed from one to other. It was endless email threads which can be hardly tracked.

At this point we decided to install our first revision control system to make audit of changes - Microsoft Source Safe. It was a real pain in the a*s to use it. It was eventually replaced by SVN which is better, but in bigger teams we had a branch merge hell with it.

Later on we also installed simple and intuitive bug tracking system which we are still using - FogBugz. We have .NET stack here for 80% of our products, that was the main reason for using these tools.

We had dedicated QA team which did only “monkey testing”. We had no automation at all. No unit tests, integration test, test automation, performance test, load test. Nobody invested time in peer code reviews. Logs were a mess, same with monitoring. It was very hard to find and identify the source of the problem.

Our development was done on personal computers and we used them as our environments as well. Shared resources. Each team member could break others environment with his change.

No REST APIs at all, instead DLLs everywhere we need to communicate with other internal components.

Product lead R&D but not the opposite. Developers has almost no power to decide.

We worked in waterfall software development methodology although it wasn’t formal. Our releases was very long and risky. It took months to release following by weeks to fix bugs of such big release.

And so on and on, you get the picture, right?

It was a nightmare…

In the next post I’m going to describe the current R&D situation in eToro.

With this post I’m starting a series of DevOps related posts. What is it, the past, the present and the future of DevOps in eToro R&D.

Lets start with basics and then drill down and expand to more details.

What is DevOps?

A formal definition taken from Wikipedia:

DevOps (a portmanteau of development and operations) is a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) professionals. DevOps is a response to the interdependence of software development and IT operations. It aims to help an organization rapidly produce software products and services.

Although it is a fine but “dry” definition, I like the more simple one found in this presentation online:

DevOps is an approach to bridge the gap between agile software Development and Operations (IT).

I’m not going to write here anymore about DevOps as you can just find below my short DevOps high-level presentation that I wrote and presented internally for eToro R&D employees last week:

Hope you could understand from this presentation what is DevOps in general and what does it suppose to give us after implementation.

We have a very long way until it’s going to be implemented, but we are on the right path.

In the next post I’m going to describe in more details about eToro past and how everything has started.

Keep on reading and as always, I will be glad to get your feedback. ;)

This is a very first post in my new minimalistic geek blog based on Octopress framework for Jekyll blogging.

It is supposed to be simple and easy to maintain. It supports markdown writing only, has no Admin section and you have to commit changes to GitHub in order to publish a new post. So if I make a spelling mistake or type, you can fork the repository, fix it and make a pull request. ;)

I used to have several blogs in the past. One was a photography blog based on Wordpress, another photo blog in Tumblr and the last one based on BlogEngine.Net with some own custom modifications was my personal blog.

All blogs I used to have, were in Russian only. It is my first attempt to write a blog in English. It supposed to serve all my English, Russian and Hebrew speaking friends.

I’m going to write here about the things that interest me and may be interesting to read for others as well. I’m going to share here stuff related to my workplace and hobbies. It is important for me to get feedback. So if you read a post and you care or have something to say about it, please comment or share with others. You can also subscribe to this blog RSS feed to get updates.

I have to stop talking to myself as nobody read first blog posts… :)