Hey guys,
I’m so happy to see people utilizing pylinac; I hope it’s helping you out in your clinic. I wanted to share some things I’ve been learning and how it can help image analysis. I’ve been reading up on and digging into machine learning. I just finished Andrew Ng’s excellent course on the subject. There are several ways I think this could be implementing into medical physics. One way is through automatic image classification. So, for fun, I trained a support vector machine (SVM) to classify four types of images I have quite a few of: picket fence (both 0 and 90), leeds (all rotation angles), pipspro, and winston lutz. Using my archive of images I did a 80/20 test/train split. After only a few minutes of training, the SVM was ready to go and it predicted the test images with >95% accuracy.
This analysis was pretty fascinating to me. Although image classification is not a new problem, I really haven’t seen it applied to medical physics. There are lots of potential applications, but one immediate one is this: automatic image classification when using the pylinac watcher. Instead of using file name keywords, imagine just pushing the images over without any intervention. The image is automatically classified and based on the classification is analyzed appropriately! This is just the tip of the iceberg. I’ve put a lot of work into identifying which slices are which in a CBCT scan. Imagine instead that the machine learning model can classify the CBCT slice into the HU slices, uniformity slice, etc without tons of rule-based searching (this excites me more than you; the end result is the same). The same principle applies for even further refinement of the above images: what if the model could tell whether the picket fence was shot at collimator 0 vs 90? What if it could know the rotation angle of the leeds phantom? This is actually possible and would reduce huge code bases to modest ones.
I don’t know if it’s time to incorporate machine learning into pylinac, but it would be unique because I don’t know of any other QA analysis software that can do that. What do you guys think?
Do it! It would be great to see an application of machine learning to our field. Plus it would helpful in understanding what the best ML algorithms are for image analysis.
What would be interesting to know is:
How much longer does it take to set up the model for training (how many images to get a good fitted model)? (from the user and developer perspective)
How different are the models between image types? Do some work better than others?
What is the fallback if the image is not classified correctly?
Would you consider making this optional in case a user wants to manually specify the image orientation / slice type? Also in this vein, to slim the setup.py requirements, using scikit-learn as an optional dependency for those that may not need it? I would tend to think most people would be glad to use it, but maybe there are some who want to drive the process themselves.
Thanks for all your hard work (and in the future on this - it’s exciting)!
I agree. If you have the time and interest, it would be a great area to get into.
I’ve been following the idea of machine learning with great interest lately, I think it is going to revolutionize so many things over the next couple of decades (and has already affected many things). You’ve already identified some wins that you could get with pylinac, so if its something that can be incorporated easily, I’d say go for it.
And thanks for the course link, I’m going to have a good look at that!
Adit,
Good questions. There are several answers to each question, but the summary is this: Training time depends on the learning algorithm (SVM, random forest, neural network, etc), the descent algorithm (SGD, momentum, etc), the number of samples you have, the number of features for each sample, and whether you’re using the CPU or GPU. Although it’s pretty deep and I haven’t wrapped my head around everything, I found this blog post to be super interesting in regard to convolutional neural networks which are good for image recognition. I have yet to use complicated neural networks as a simple random forest and SVM have proven accurate enough so far. Additionally, since scikit-learn is easy to use and is easy to install I’d probably stick with that for easy portability. I stayed up till 2am last week trying to get my GPU connected to more complicated libraries, which I did and is orders of magnitude faster than the CPU, but I wouldn’t wish that install process on anyone. Although it takes a while to train the model, once it’s trained it can be pickled and shipped with a package and it’s quite quick for individual classification. Unfortunately, models can’t be tuned later on by the user.
Fallback is an issue. I don’t know of a good way of dealing with inaccurate classification. The current method of filenames or some sort of keyword is a good way, but that sort of defeats the purpose. More experimentation improving the accuracy may help. The good news is the classification is pretty clear. E.g. there’s never a mixture of a leeds phantom and picket fence.
I definitely would want to make the classification optional. Given that the watcher script now has a configuration file and options, the ML classification option could be a flag the user sets.
And Darcy, I fully agree and think we’ve only started to conceive how we could use machine learning in our field. There’s a student here at MD Anderson who’s using ML to predict contour expansions; pretty exciting! Another area I’d love to investigate that is unrelated is anomaly detection (where outliers are rare or you don’t have any yet) which could be used for plan checks. Pass in your plan with all the various features and make sure it’s “normal”. Although rule-based checks are probably just fine for this, the idea can be applied to any feature or level of complexity (T-staging, MU, age, etc).