▲ 139 points
back
126 comments
My first job out of college was working at Epic on MyChart. Great people, terrible code.
Epic’s main problem is a lack of clear internal code ownership. Everyone owns all the code. This means that even if you clean something up, someone on the other side of the company may come in and mess things up again.
This led to really defensive programming where developers would never refactor, they would simply add a new if case for their new functionality somewhere deep in the code, then prop drill the data down. This led to every core function having over a dozen parameters and hundreds of branches. It eventually became impossible to reason about. Cross team calls were just function calls rather than defined apis. This made it fast to develop code initially, but terrible to own long term. This mainly applies to their Mumps code.
While I was there I felt like Epic was beyond saving, but with a big push there may be something they can do:
1. Enforce some level of code complexity. Best practice is 40 lines per function and no more than 4 parameters per function. Epic probably shouldn’t shoot for that, but a 100 line limit and 6 parameters per function would already be a huge improvement.
2. Enforce strong code ownership. Epic has many people who are there for life, let them cook. Epic should segment off code to certain teams so those owners can fix it at their leisure. Cross team api calls should be clear API contracts. It would require some more discussions to get feature requests approved since not everyone can do anything anymore, but the code would gradually improve.
Epic is too important to fail. I hope things have started to improve since I left.
Most places I’ve worked including Uber and Airbnb allow teams to contribute code to other teams services. Quality is protected by requiring “blocking reviewers” on pull requests. Blocking reviewers requires one person from the services team to approve the pull request.
I think this is better than requiring teams to make all changes themselves which slows things down significantly considering each team has their own roadmap and priorities
> 1. Enforce some level of code complexity. Best practice is 40 lines per function and no more than 4 parameters per function. Epic probably shouldn’t shoot for that, but a 100 line limit and 6 parameters per function would already be a huge improvement.
If I were to enforce some kind of arbitrary code complexity threshold for functions I would put a cap on the limit of possible of branching combinations based on parameters within the code. Like around 16 (branching combinations are exponential).
For example a function with 20 parameters but only one if statement is fine. A function with 5 parameters but several nested if statements is not.
What languages is their codebase developed in?
Ha, we were pushing complexity metrics back in 2012. The Security team added it as a PQA step in 2013 when I was there, no idea if that stuck around.
How many tech companies have been consistently crushing it in their fields for nearly 50 years? I didn't appreciate Epic enough while I worked there (left 12 years ago), but I did learn some of their ways:
- There were no PMs. No one was chasing vanity metrics. No endless barrage of a/b tests. No growth hacking. Instead, product was built the old-fashioned way - by talking to customers; quite often, customers would reach out to us! "Please build time-saving feature x", "support new medical procedure y", "help us publish more research by analyzing z". The heap of ideas was large, and teams were free to apply their own ranking functions. Some top-down strategic initiatives were threaded through all products. This led to every release being packed with things that customers wanted.
- They leveraged their advantages. Plentiful, cheap land → everyone had an office with a door, possibly shared with 1 other person. Productive people flourished. On the flip side, those who were languishing fell faster and deeper into their holes.
- They learned to live with their weaknesses. Not everyone wants to live in WI, and many who are willing to try end up looking for warmer pastures in a few years. Epic ended up being an early-career transit hub. Their attrition rates would cripple most tech cos, but they shored themselves through extensive training programs, and by rewarding the anchors that stayed.
- They took pride in the product. Every month, the CEO personally presented the latest qualitative assessment of each product (as assessed by an independent third-party). If your product slipped from green to yellow, the pressure would trickle down to you. Also, there were no outbound sales, because having the product speak for itself was the sales strategy.
Of course, not everything was rosy, but much has been said here about the shortcomings of Epic, and I wanted to point out what's contributed to the immense stamina that has kept them in the lead for decades.
My wife is a doctor and has a lot of negative things to say about Epic. I think they have succeeded due to vendor capture and lock-in in spite of their shoddy software that doesn't actually meet the needs of the users. Every doctor I've met through my wife would ditch it in a heartbeat if they could.
>- Instead, product was built the old-fashioned way - by talking to customers; quite often, customers would reach out to us! "Please build time-saving feature x", "support new medical procedure y", "help us publish more research by analyzing z".
Think that might be a plus. PM/PO has ruined the industry. This way at least one has a direction to the customers, which is something I can't say about large companies.
There’s also a strong culture of ownership of quality for individual developers. Design documents are expected for even minor changes, and each change goes through at least 2 rounds of well-documented testing by 2 other developers on your team. Compared to other companies where automated testing is the norm, it’s startling how few bugs they manage to ship.
Another thing I thought was interesting given modern dev practices is that you don’t even touch production code until after about 6 months of training, including exams and testing on the actual functionality of the product, even bits you’ll never interact with personally. They’re serious about making sure you know what you’re doing before you get within a mile of something that could affect patients.
They also have a pretty strong accountability culture. I shipped two fairly embarrassing bugs my first year and had a very serious conversation with my manager about whether I should continue to be employed since I’d used up about 50% of our team’s allowed bugs for the year. But on the other side, once I got my feet under me, they were very good about recognizing improvement.
No PMs is the real growth hack.
I did an internship at Epic and didn't fully appreciate it at the time, but I think their CEO Judy might be one of the best tech/general leaders ever.
Epic has a reputation of hiring lots of new college grads. For software engineers that's not exactly uncommon, but Epic actually has a lot of employees working under titles like technical solutions/implementation solutions (or something like that): the people directly supporting the hospitals using Epic. Because these are pretty specialized roles, Epic has a very formal and fleshed out training program for their new hires with classes and courses and such, and it can take months to complete. They not only have their giant campus in Verona, they have an entire training center there, a huge auditorium for allhands, and a very streamlined recruitment process (for a ~21 year old it feels over the top luxurious). Although Epic does hire from more selective schools it seemed the majority of their new employees are from state schools in the Midwest.
They also are private despite their size, just basically don't do M&A and are, relatively speaking compared to other big corporates/tech companies, in the middle of nowhere.
In corporate America this is a highly unusual way to operate. I think it's underrated how big of a "risk" all these heterodox corporate strategies are for an executive and it speaks to amazing ideation and execution on Judy's part. Also, even though Epic does have a decent amount of turnover, she has taken a chance on tens if not hundreds of thousands of young people who didn't have the skills she needed them to have already, by giving them months of training and a really solid start to their careers.
> it seemed the majority of their new employees are from state schools in the Midwest.
I went to the University of Wisconsin for grad school, and I knew lots of people there who went to work for Epic.
I confess that, before I applied, I imagined that Wisconsin was nothing but cow pastures and football. I imagine this sentiment is shared my many young job-seekers outside the Midwest. Now, having lived there for four years, I have to say that UW is an amazing university, and Madison is my favorite city in the US.
The link is either down or blocked outside of US.
For those not from US this has nothing to do with EPIC Games and I assume no part of Unreal is using MUMPS for those who are reading comments before they click on the link.
This Epic has something to do with healthcare software. And kind of surprised how many people who used to work there appeared on HN.
For what it's worth, am an American and also thought this was about the gane company from the headline.
Epic is used extensively in the UK, Europe and Australia so to say it’s a regional thing seems odd? They’re speculated to be one of the largest privately held companies in the world.
You are surprised because you don’t quite conceive of how large of a footprint they have grown in the US healthcare system. All the boomers and genxers who used to be on MySpace are now on MyChart.
Europe here, works for me.
I interviewed at Epic for my first job out of college a decade ago: while the campus is indeed beautiful, the sense I got was that they were trying to emulate Google's quirkyness while offering much lower salaries (but still relatively good given the CoL) and a less exciting product domain. I'm not sure how well that quirkyness appeals to prospective applicants in 2025.
Like many here, I worked at Epic just out of college and left after a few months. Everything others have said is true. I call the campus "Disneyland for sad people," because it's gorgeous, but also totally artificial, and nobody is happy.
The one great thing Epic did for me was get me to Madison, WI, an amazing city of great people where I found a much better job and stayed for many years. I still miss it sometimes.
Epic used to be a heavy user of the legendary/infamous MUMPS programming language. I wonder how much that’s still being actively developed.
This 2007 classic explains how a case of MUMPS progresses when you’re a programmer:
The fact that Epic remains the best available solution in healthcare is quite sad. That entire industry is rotten to the core.
Since this has become a general thread about Epic, here are my comments:
I am amazed at some of the software Epic has built for itself over the years. Using its own database product (the backbone of the product they ship to customers), they built their own code review tools, design doc review tools, project management tools, time logging tools, etc. There is a unity and cohesion to the process of getting things done at Epic, better than my experiences at big tech.
It is very easy to answer questions like "how many dev-hours were spent fixing bugs caused by the code written to implement project X?" or "will there be any days next week where every dev who has contributed to codebase Y will be out of office?"
Imo they could really benefit from staffing infra/tooling teams better. Too many product devs, not enough devs tackling the low-hanging fruit that would make product devs way more productive.
As a patient, I seek out MyChart because it's really well put together from my perspective. I've no idea how medical professionals and administrators feel about it, but personally I've had a great experience. I saw someone from Epic was here, so I just wanted to say keep up the good work :)
I'm guessing somebody was listening to the latest episode of Acquired?
The significance of this to me is by contrast to most of the valley companies (FAANG and their offshoots). Over the last 20+ years, people have been trying to build a mission-oriented company with a great culture, and confident enough to build their own tools. That was the default story.
It turns out companies are transient or have been internally infiltrated by such (outsourcing- and ambition-driven) politics that any mission is more supplement than reality, and there's no sense of controlling your own destiny.
So perhaps the dream persists out in the tech boonies in the ultra-sticky EHR domain, goosed by the Obama/insurance mandates to digitize, where developers are trapped by unportable skills. (Or perhaps in smaller B2B companies filling a niche.)
Ayyyy, my day job is on HN. The campus is indeed cool, and I like working here (though I haven't been here nearly as long as some people!)
If anyone's interested in Epic and wants one employee's opinion, my email's in my profile.
My first post-college job! Nothing but good things to say about it, all my colleagues were whip smart at what they did. I especially liked the interview process, where I had to do a couple of standardized tests online to prove I was in the top x% of test takers. Given that I had to lock down a full time job as fast as possible after college it was a real time saver to just be able to demonstrate objective general competence like that and move right on to the interesting stuff.
I've long since moved into startups, but Epic was a great place to learn process and the less technical aspects of enterprise software development. The tech has modernized a lot in recent years, but still isn't transferable quite as easily as other tech companies - but that legacy means it's an amazing place to learn the why behind best practices.
Funny story about a visitor to a game development office.
About 8 years ago I was working on a mobile game where you could purchase specialized dragons and eggs. Some of these could be pretty expensive, but since they were high end items we wrote special GPU shader code for them so they had cool special effects on them. We tested these as well as we could -- we had a room with maybe 100 or so mobile devices -- but of course we couldn't test on everything.
One day an irate older lady came to our office, and our receptionist for some reason let her in (probably thinking old lady = harmless?). Keep in mind our office was unlisted because we didn't want fans dropping by. She had driven all the way up from Arizona to Colorado (although I don't think it was the only reason she drove up), and she accused us of ripping her off, because she had bought one of these fancy dragons and instead of getting what she saw in the promo materials, its wings were black! I didn't hear or see this directly, instead it was the main topic on our Slack chat with everyone being cautioned to Play It Cool.
I didn't think much of it until I realized it was _my_ code that had caused this entire issue in the first place!
Luckily we had a really good customer service guy that defused the entire situation, but that's the first and hopefully only time I've been tracked down in person by a customer for a bug.
For those wondering, this is way more fun than the Epic Games office
Wisconsin local, currently attending UW-Madison for CS, and visited the Epic Campus when I was going to a local high school less than 30 minutes out. Talking to the engineers and exploring the campus is what solidified my choice in deciding to become a software engineer, so thank you Epic!
Ah yes. The company that wanted to know my SAT and GRE scores, and then required me to do a personality profile quiz before rejecting me (did not even get to the entrance exam, which I was looking forward to).
Still, I heard working there was quite good. Obviously not FAANG level salaries, but after you left and completed the 1 year non-compete, other health care companies and/or hospitals would pay a good premium for your MUMPS expertise.
(None of the above is sarcasm, BTW)
Is there some significance to this page other than the funky cow theme/elaborate travel guide?
Just because it’s nice to visit doesn’t mean you should spend a year of your life there.
I consulted with Epic once and got to visit the campus.
Incredible place, super detailed, and I loved the cafeteria setup they had (great food too.)
I definitely got a feeling that folks got burned out pretty quickly though.
I read all these comments thinking it was about Epic Games. Then after 5 minutes clicked the link and discovered a one of a kind American company.
I see jobs posted for epic all the time, I just have no interest in relocating to Madison and they are vehemently against remote work
I was excited to see the domain but it turns out this is Epic the healthcare software company not the game developer.
In case you're wondering what the Epic campus is like, this 18-minute minute tour by an Epic engineer will give you some idea. Well worth a watch!
Side note: the real Verona seems to me much, much better and nicer than the one in the US, or the Epic campus.
> Emerald City
> This extraordinary building is a playful tribute to the timeless movie
Lyman Frank Baum would be surprised to read this :)
What’s a typical Epic installation. Does it run on a single server? A cluster?
so, how much lower than California are the salaries in Wisconsin?
doc first, old linux hacker second
Epic sucks. It only sucks marginally less than all the others. As a corp, they are expert at abusing mechanisms for lock-in as well as network effects
If you all were forced to use tools as shitty as the EHR's, no one would be a software engineer
A re-write of VistA would be the way to go, but someone paid off someone, and the VA is doing a disaster of a changeover to Oracle's Cerner
tl;dr Healthcare software is a steaming pile of shit
Epic is known locally as an exploitative, abusive employer of software engineers. Work-life balance is poor, pay is mediocre for the industry, and skills with their in-house tools don't transfer outside Epic. They have an extensive non-compete clause with EXTREMELY aggressive enforcement:
https://isthmus.com/news/cover-story/opportunity-lost-epic-n...
They're also vehemently opposed to remote work, to the point that during COVID they tried to force employees back into the office in August, 2020 (!) in violation of a county public health order (!!!):
https://www.wpr.org/economy/workers-officials-urge-remote-wo...
Epic's Glassdoor reviews are terrible. Several personal friends each lasted less than a year at Epic out of college before finding new, better-paying employment elsewhere. Since Epic is privately owned and its founder and CEO has stated she'll never sell, its corporate culture will never change. It's better than no job at all but if you have other options, avoid.
Must every company in the Midwest use a cow theme in some manner?
Staff use cow bikes, cow carts, or cow vans to mooove across campus.
It was cute when Gateway did it, still cute when FatCow does it (it is in their name), gettin’ a little cringe for the late-comers, though.