I love this story for the obvious reasons but also
1. Imagine how horrible it'd be doing tech support for the pentagon?
2. I've definitely been in situations where deploying nuclear weapons on the tech support of a subcontractor would have been a proportional response https://twitter.com/syskill/status/1191460529606410240
For #1, imagine they call up and say their Foobinator isn't working.
"OK, what are you trying to do with it?"
"we can't tell you that."
"ok... can we send someone to look at it?"
"No. It's too classified to look at."
as for #2... there was a particular subcontractor that took so much government money and delivered so little value that if we'd just started the 10-year project we worked on them with by dropping a nuke on a certain west virginian town, it would have saved us money
"yeah we had to spend 20 million to rebuild that town and another 5 million to another contractor to build the site, but if we'd let them do it... god. it would have been ten years at a couple million a year, and we'd still have a shity useless non-site at the end of it"
so at one point , we were discussing the year long process to extract our data from them after having to sue them for massive payroll fraud and they were threatening to delete 15 terabytes of our data it took 10 years to collect, thanks to some clever lawyering on their part...
and it was estimated it was going to take another YEAR to write a website to replace the one they had been developing for the past 5.

That pissed me off so much that I went home that weekend and recreated their entire website from scratch.
my version ended up both performing better and being more functional, and it was running off my eeepc netbook instead of 3-7 servers.
this was the same job where tech support got involved in another way: At one point the head of IT suggested to my boss during a meeting that the easiest way to solve the issues we were facing (the ones IT was causing) was for me to be temporarily reassigned to the IT department
as a contractor myself that was certainly possible, and it'd mean they could give me free access to the securely gated IT area. It was very secure:
You had to come to the double doors, face the camera, and use the provided (red!) phone to call the security guard on duty.
it would have been the most secure location in any other building, but since this was a government office there was also an FBI interrogation room and a double-door "airlock" of bulletproof glass for the IRS office.
my response to "let's just move Foone to IT!" was something along the lines of "if you do that, I will quit, my contractor agency will get in trouble, and oh yeah, approximately 150 million dollars of taxpayer money will be wasted as an entire decade's worth of work gets deleted"
that was during the meeting. after the meeting ended, the head of IT asked "if I had a negative impression of IT", and I pointed out I'd gotten JOB OFFERS purely in reaction to me complaining about how bad IT was at this job.
Don't work for the federal government, kids.
They can't (and don't try to) pay you enough to possibly offset the amount of damage done to your soul.
at the point this was happening, I'd already gotten an offer to work at another company, and that company was going to pay me nearly 3 times as much as I was making at the government. But I turned it down, at the time, because there was this ongoing problem with the subcontractor
The (lack of a) federal budget had just destroyed my department. Everyone but me and two others (who were both very senior management who'd been there 15 years or more) had lost their job, and the sole reason I stayed on was because I was the only hope to save the data.
if I'd left right then, one of two very bad things would have happened:
1. Everything the department had been doing since their creation would have been lost. 10 years worth of work, the salaries of a couple dozen employees, and 50 million scanned documents: GONE.
2. They'd have to surrender to the demands of the subcontractor they were suing for massive payroll fraud, and work out some other way to preserve their data. That would mean continuing to pay a bunch of conmen who had been ripping off the government for years.
It would have been far better for my own career to say "that sucks guys, good luck!" and peace out to another job that anyone could do and would pay me many more dollars.
But I went with my heart and not my wallet. That was almost certainly a terrible, terrible move.
but the upside is: I did it. I rescued all the data before the deadline of cutting off contact with the conmen. Nothing was lost, despite their best efforts. The replacement site was built, and access continues to this day.
Getting to that point was a massive headache, sure, but it's done.
They took advantage of how government budgets work and how we couldn't just drop the cash to get 15 tb worth of hard drives and have them mail them back, and instead got two, and cycled them between us
those drives were only 3tb each, so it ended up taking about 9 back-and-forth mailings to get all the data.
And they did hard drives for a reason: IT security naturally freaked out about bringing in an entire hard drive of data to plug into a server, so they had to do virus scans
which took a few days, and then we'd possible find out at the end of the virus scan that the data was corrupt, and we'd have to mail it back. All plans to eat into our limited time before the "DELETE EVERYTHING" deadline looming in the near future
so we got IT to give me SPECIAL EMERGENCY ACCESS into the IT area, where I was given a special burner laptop to access the external drives before the virus scan.
This laptop had no network access, and I was allowed to bring in a CD but not bring anything out.
so I could bring in a couple python scripts I'd written to test their database dumps (exported with their homegrown and broken CSV implementation because OF COURSE WHY NOT) and verify that the metadata was correct and all the files were there.
and then I'd hand in my CD, which they threw away, and the laptop was immediately re-imaged to destroy any possible viruses.
IT-security could then start the multi-day process of scanning the data for viruses.
there was no way to bypass the virus checking, but it was just hilarious in context.
For one thing, we'd had a inter-network connection set up with this subcontractor for over 5 years: they could send data right into our network, and us into theirs.
if their network was full of viruses, WE ALREADY HAD THEM.

But the worst part? this was the FIRST time we virus scanned this data: it wasn't the last.
because while the original plan was "they send us some hard drives, we virus scan them, then plug them into a server", that ended up being impossible, because of two simple factors:
1. NTFS
2. Redhat.
It turned out they had formatted all the hard drives with NTFS.
That makes sense, they were 3TB drives. ExFAT didn't exist yet, they used all Windows systems so ext3 or whatever was out of the question, and 3TB is a bit too big for FAT (usually)
but we were an all-unix shop (which was hilarious for another reason: all developers had to use windows, and VMs were not allowed) so all our servers were Redhat or Solaris or HP-UX.

And how many of them supported NTFS? It turns out the answer is "zero"
Now you might think "hang on, hasn't linux supported NTFS in one form or another for like... ever?"
Kinda! NTFS-3G with full read/write has been around sicne 2007, and NTFS had starting being added to linux (in read-only form) since 1999 with the release of 2.2
but this was 2011, so naturally the government hadn't updated to such new things yet. We were still on ancient versions of redhat, which either predated the ntfs modules or had them disabled as they weren't stable yet.
so in the end, do you know how all 15tb of data got moved off those hard drives?

on my Dell desktop.
I plugged them into the front panel USB and copied them up using an FTP client.
which means that the files got scanned again when they went through my PC (because of course it had a virus scanner active: ask me sometime why I couldn't print at 8:30am or 2:30pm)!
and guess what: when the FTP connection goes to the server, it goes through the network firewall, which SCANS IT AGAIN
so yeah... the whole reason this confrontation happened with IT was because they wanted to pre-scan all the hard drives coming in from a subcontractor we had implicitly trusted for 10 years, and not rely on the OTHER TWO TIMES IT WOULD END UP GETTING SCANNED.
BTW, in the end it turned out the files got scanned a 4th time:
The FBI showed up a while later and had crafted a special regex to try to locate a spear-phishing attack against our director's office. They didn't find anything, but did try to get me fired. That was a Fun Day.
Technically I'm not supposed to know that they were doing "virus scanning" with a regex, but apparently even TOP CLASS FBI HACKER DUDES don't realize that if you search a bunch of files with pcregrep the pattern you're searching for ends up in the output of "ps aux".
but yeah. We had to talk to the subcontractor during special speakerphone meetings with lawyers from both sides present, we weren't allowed to send emails to them, and all this while we were trying to get a bunch of data from them under a deadline.
They did everything they could to slow us down and get the deadline overshot, since then we'd have no choice but to pay them more.
But they lost. We got the data. They got 0 more cents out of us.
and all it cost was MY FUCKING SANITY
my favorite stupid incident is one of the times when I was scanning files on the Burner Laptop, it was getting towards the end of the day. I wanted to turn it in when it finished, so I was working late. (I'd brought a library book, because no wifi)
I went out to grab dinner.
and I came back around 5:20 and they wouldn't let me back in.
I'd been in the IT room since around 10am that morning, but they said I couldn't come in because I wasn't "on the list"
it turns out that IT's security people had two separate lists of people they maintained to allow people in: regular and after-hours.
And since I was out at 5pm, the security guard had changed, and was now looking only at the after-hours list.
I pointed out that I had a special cubicle set up in there and had been working there all day, but that didn't matter.

So I asked if I could get my book, at the very least. They said no.
and the "No, you can't have your book" is the perfect microcosm of the entire attitude of the IT at that place.
does this harm anyone? no. is it a security risk? no. would it take them any serious amount of time to do? no, of course not.
would it help me? certainly.

Fuck ya, you're not getting this book.
Thank god I'd taken my keys with me or I'd have been locked out of my car and apartment, too.
These are the same fuckers who maintained the strict list of software that was allowed to be installed on developer machines, and when they got overruled into allowing an instant messaging client for developers, retaliated a few days later...
They decided that while the web-development group had gone over their head and gotten the upper management to force them to allow installing Pidgin for developers... they hadn't approved GTK, the library on which Pidgin is built.
So after the developers had been given a few days to try out the new IM system, they then decided it was "not approved" and responded by RE-IMAGING ALL THE AFFECTED MACHINES.
They were the ones who installed GTK & Pidgin. They could have simply admined in and uninstalled them.
But it wasn't about removing GTK, or Pidgin. It was about sending a message.
And the message was clearly received: Don't even think about going over IT's head or they can and will ruin your ability to do your job.
I sure hope you had everything you were working on uploaded into subversion, or who knows how many days of work you'd lose?
on top of all the days you lost anyway, reinstalling apps and reconfiguring IDEs and SSH & FTP clients.
We never got our IM client.
No one pushed it again.
BTW, the reason we were on ancient redhat?
It was also IT's fault, but in a weird way: Threatened retirements.
see the upgrades were possible but it would have required some mandatory retraining to move to a newer version.
And certain senior sysadmins in IT didn't want to go to training, so they'd made it clear: You can't upgrade without losing them.
And they'd done this because while government employees have rules about how much warning you need to give before leaving a job, because of how they work on contracts, they also have a lot of leave time that they can stockpile.
and at some point the amount of leave stockpiled exceeds the amount of notice you need to give for leaving.
So it was surprisingly common to have federal employees who'd been there for 10-20 years go "So I've just announced, I'm retiring in a few months, and my last day in the office is Friday"
And several senior IT employees had decided to play this card against the idea of ever upgrading their servers to a newer OS: If it's decided to do that, they're announcing their retirement and walking out.
So, ancient redhat it was then, at least for the foreseeable future.
This whole leave-stockpiling thing didn't work for contractors, btw. I had a coworker who had to come to my desk one day and say "hey, I can't work on that project we were planning, I'm gonna be out for 3 weeks starting Monday"
and I was like "oh, are you having surgery or going on a vacation or something?" and the answer was "nope, just contractor accountant came to me and said I haven't taken enough of my paid-time-off hours"
and the only way to solve that before the end of the year when he'd have Too Many was to just take most of November off.
This happened because any stockpiled PTO hours legally had to be paid back to the employee when they quit or were fired, so they had instituted a strict upper-limit on how many you could have at the end of the year
because at the end of the year they had to count these on their books as some kind of possible debt, as the employees could decide to quit at any time and they had to immediately pay out a bunch of money.
This was the same contractor that fell into a complete panic one weekend because it turns out that when they'd written that year's contract, they left ME out of it. And it took them three months to notice.
they weren't actually worried about losing me, of course.
They were worried that they'd been paying me for three months and the government wasn't paying them for my position.
so they were potentially in three months salary worth of debt over my position, because they'd screwed up the contract.
contracting positions had gotten extra complicated that year because they had to become a subcontractor to another contractor and transfer 51% of their employees to the other contractor, and also my entire division had ceased to exist the year before.
but yeah. I got called in on a random Friday in March and they were like "oh hey, it turns out we don't actually have a position for you in the contract. So unless we can figure out one real quick, Monday will be your last day"
(It was)
fun fact: They never told the government that they fucked up on this position.
I know this cause I happened to talk to one of the federal employees later and found out the contractor told them that I "quit".
I also know this cause a few months later I got a job notification for the contractor (through a hiring company) that appeared to have been literally written based on all the things I'd built while I was at the government office.
it was like:
JOB ROLES INCLUDE:
1. maintain a website for serving 20 million PDFs
2. maintain a database for tracking inventory of paper documents in our warehouse
3. maintain an internal database for tracking 50 million scanned documents across arbitrary metadata.
and I was thinking "hmm, that's funny, that certainly sounds like the site I maintained and the databases I created while I was at this job"
Almost like they're trying to hire someone to directly replace the person they let go a few months earlier.
So I of course submit my resume, which INCLUDES ALL THOSE THINGS RIGHT ON IT, DURING MY TIME AT THIS POSITION and I get a call back from the hiring company.
They say "hey, we got your resume, and you look like a good fit, but our records say you voluntarily left this position a few months ago?"
And I told them "no, I was at this position until they ran out of money due to leaving it out of the contract, and they let me go"
"Huh, that's not what our records says, weird. I'll look into it and call you back, assuming you're still interested in this position?"
"I am!"

I never heard from them again.
at some point I'm just gonna write a book based on all the assorted bullshit that happened while I was there.
I don't know if anyone would read it, but it'll probably be good for my mental health to write it down.

Having this much angry cooped up in there can't be good.
ANYWAY, the tl;dr is don't work for the federal government.
Directly or as a contractor. Really, don't. It's not good for you.
and in theory I've got an etsy where you can buy one of my weird artwork PCBs but etsy isn't loading for me now, so hopefully that's just on my end and also that this is the right URL:
https://www.etsy.com/listing/736823648/copyprocontrol-pcb-v12
BTW, if you ever meet anyone who says they do network security for the government, ask them if they have any interesting stories.
They always do.
You can follow @Foone.
Tip: mention @twtextapp on a Twitter thread with the keyword “unroll” to get a link to it.

Latest Threads Unrolled: