Machine Learning System Design Interview

Product Information

For our recommender example, the ranking component can be built with an ML model. We can rank the candidates by their predicted outcome for the user. For example, maybe based on our initial discussions, perhaps we’re trying to increase engagement by showing posts that increase user interactions with the posts. There’s lots of ways to do this: Valerii: I think algorithms are just one of the smallest parts, 1-5%. Well, I was speaking with a candidate recently and I told him “Look, imagine that you're a machine learning engineer in the company for two years,” He said, “Okay, okay. I can imagine that.” “Imagine that you spend an immense amount of time creating an algorithm – finding the best algorithm, setting up the loss function, the metrics, all the rest. It took you a humongous amount of time – two weeks. And you’re in the company for two years. What do you do?” Right? ( 48:07)

Gradient boosted trees: Better performance than logistic regressions, can find non-linear interactions, typically doesn’t require much tuning.

Alexey: Yeah, exactly. Okay. Maybe one last question. It seems like you have a very solid data science profile, from Grandmaster at Kaggle. That's pretty solid. ( 58:35) Valerii: As soon as we have a probability, we can calculate the expected fraud, which already leads us to the first metric to assess the quality of the model, which is “expected calibration error,” or “weighted expert calibration error.” Okay, we've got that. We also know that the ideal solution would be a binary classification task – one and zero – the crystal ball, right? We know that this will never happen, however, we know that it's a binary expression and that the output has to be between zero and one and it has to be a probability. So that also tells us “What should be our loss function?” The loss function should be from a family of a proper scoring function. ( 13:58) Given a text and knowledge base, find all the entity mentions in the text (Recognize) and then link them to the corresponding correct entry in the knowledge base (Disambiguate).” This book is the result of the collective wisdom of many people who have sat on both sides of the table and who have spent a lot of time thinking about the hiring process. It was written with candidates in mind, but hiring managers who saw the early drafts told me that they found it helpful to learn how other companies are hiring, and to rethink their own process. You can also make use of other creative data collection techniques. For example, you can build a personalized experience in your product by collecting data from users. If you’re working with a system that uses visual data, such as object detectors or image segmenters, you can use GANs (generative adversarial networks) to enhance the training data. Other things to consider include:

business requirements change, and (3) data distributions constantly shift. Without an intentional designMake sure you bring up how you would launch the system and actually evaluate whether it’s achieving its business objectives. This is almost always via A/B testing, which has lots of its own nuances. Talk about which metrics you’d measure and statistical tests you’d perform for an A/B test. You can go into some depth talking about ramping patterns and issues that arise with A/B testing. Model Lifecycle Management ML aims to solve a multitude of complex problems. It has made rapid progress in areas like speech understanding, search ranking, and credit card fraud detection. Companies are leveraging these technologies across industries from healthcare and agriculture to manufacturing and retail.