Supplier Scorecards: How to Measure a Machine Shop

You already did the hard part. You vetted an overseas machine shop, ran a first article, placed a purchase order, and parts arrived. The question now is different from the one you asked six months ago. Back then you asked can this shop make my part at all? Today you need to answer is this shop getting better or worse, and how do I know before it costs me a line-down?

That second question is where most buyers go quiet. They form an opinion from the last bad shipment or the last friendly call, and that opinion drives whether the supplier keeps the business. A scorecard replaces that recency bias with a number you can defend to your boss, discuss with the shop, and watch over time. This article gives you a scorecard you can copy, the formula behind every metric, and what a good versus a bad number actually looks like on a machine shop.

Vetting qualifies once; a scorecard measures forever

These are two different jobs and conflating them is the first mistake. Vetting is a pre-order gate: you look at certifications, sample parts, capacity, financial stability and communication before you commit money. We covered that end to end in how to vet an overseas CNC supplier, and this piece assumes you have done it. Vetting answers a yes/no question at a single point in time.

A scorecard is the opposite: it assumes the answer was already yes, and it watches what happens next. A shop that aced your audit can drift. A new production planner joins, a key CNC operator leaves, the shop wins a big customer and your small job slides down the queue. None of that shows up in the audit you ran last year. It shows up in your on-time rate creeping from 97% to 88% over four months — but only if you are recording it. Do not repeat the vetting checklist here; reference it, and move on to measurement.

The practical rule: run vetting once (and re-run the full audit maybe every two to three years, or after a major change at the shop). Run the scorecard every single month the supplier is active. One is a photograph, the other is a heart-rate monitor.

The seven metrics that actually tell you something

You can measure a hundred things about a supplier. Most of them are noise. These seven cover delivery, quality, responsiveness and paperwork — the four ways a machine shop makes or ruins your month. For each, I give the formula, the data source, and a realistic good/watch/bad band. Treat the bands as starting points and tighten them to your own tolerance for pain.

1. On-time delivery rate (OTD)

The single most predictive metric of a supplier relationship. Calculate it as (number of line items delivered on or before the confirmed ship date) ÷ (total line items shipped) × 100, measured per month. Use the confirmed date on the order acknowledgement, not the date you wished for, and not a date the shop quietly revised after the fact — lock the baseline when the PO is acknowledged.

Decide up front whether you measure ship date (parts left the dock) or receipt date (parts arrived at you). For an overseas supplier, ship date is fairer to the shop because freight and customs are outside their control; receipt date is what your own production line actually feels. Many buyers track both and hold the supplier to ship date while watching receipt date for logistics problems. A good OTD is ≥ 95%. Between 90 and 95% is a watch zone. Below 90% and you are building safety stock to cover a supplier who cannot be planned around, which is a hidden cost you are paying whether you name it or not.

2. First-pass yield and defect rate

First-pass yield (FPY) is (parts accepted at incoming inspection with no rework) ÷ (total parts received) × 100. Its inverse is your defect rate. For low-volume, high-value CNC work, express defects as a percentage. For higher volumes, convert to parts per million: PPM = (defective parts ÷ total parts) × 1,000,000. PPM sounds like automotive jargon but it is just a way to make small percentages legible — a 0.3% defect rate is 3,000 PPM, and "3,000" is easier to trend on a chart than "0.3".

A mature machine shop on repeat parts should sit at ≥ 98% FPY. New parts and first articles will always be worse and that is fine — segment first-article results out of your production FPY, or you will punish a shop for the learning curve every design deserves. Watch the trend more than the absolute number: an FPY sliding from 99% to 96% over three months is a warning even though 96% still sounds acceptable.

3. Non-conformance (NCR) rate and closure

An NCR is raised whenever parts do not meet the drawing or spec. The count matters less than two things: how long the shop takes to close an NCR, and whether the same failure comes back. Track average closure time in days and count repeat NCRs against the same characteristic. A shop that raises its own NCRs, tells you before you find the parts, and comes back with a real root cause and corrective action (an 8D or equivalent) is more trustworthy than a shop that never reports one — the second shop is not defect-free, it is hiding defects. Reward transparency in how you score this, or you teach your supplier to go quiet.

4. RFQ-to-quote time and quote accuracy

Two sub-metrics. RFQ-to-quote time is the clock from when you send a complete CAD package to when a real number lands in your inbox. This is a genuine differentiator: some shops take three to five days, which stalls your own quoting to your customer. Sendot commits to a quote within 12 hours of receiving complete CAD, and you should hold any supplier to a stated, consistent turnaround. Quote accuracy is how often the final invoice matches the quote without surprise charges: (orders invoiced within tolerance of quote) ÷ (total orders) × 100, where you define tolerance (say, within 5%). A shop that quotes low and true-up later is not cheaper, it is unpredictable, and unpredictable is expensive when you have already priced the job to your own customer.

5. Communication responsiveness

Soft to measure, hard to live without. The simplest proxy is average first-response time to an email during the supplier's working hours — remember an overseas shop like Sendot runs on China time (UTC+8), so a 14-hour offset from US Eastern is structural, not rudeness. Judge them against their own business day, not your clock. A workable target is a substantive reply within one business day. Score it low if you are chasing answers, escalating through multiple people, or getting "OK" replies that do not actually answer the question.

6. Documentation completeness

Every order should arrive with the paperwork you agreed to — typically the FAI report for new or changed parts, CMM inspection data on critical dimensions, and material certificates. The metric is blunt: (orders with complete, correct documentation delivered without you asking) ÷ (total orders) × 100. The phrase without asking is the whole point. A shop that can produce a material cert when you chase it three times is not the same as a shop that ships it in the box every time. For regulated buyers this metric can outrank everything else.

7. Cost and commercial behavior

Not raw price — you chose the shop knowing its price. Track price stability (are quotes drifting up without a material-cost reason?), willingness to hold pricing across a blanket order, and how they handle engineering changes. This is a lighter-weight metric than the first six; keep it on the card so cost discipline is visible, but do not let a low price paper over a bad OTD or a rising defect rate.

Why weighting matters more than the metrics

Here is the part most templates get wrong: they hand you a scorecard with equal weights and call it done. Equal weighting is a decision you did not make on purpose, and it is almost always wrong for your specific situation. The weights are the strategy.

A medical-device buyer whose supply chain lives and dies on traceability should weight documentation completeness and NCR discipline heavily — a missing material cert can stop a lot release cold, regardless of how fast the parts shipped. A high-volume production buyer feeding an assembly line should weight OTD and first-pass yield above almost everything, because a late or defective shipment stops the line and the cost of that dwarfs a slow quote. A hardware startup racing to a demo might weight RFQ-to-quote time and communication highest, because speed and clarity are what keep the project moving.

The mechanic: assign each category a percentage weight that sums to 100. Score each metric 1–5 (or 0–100) against the bands you defined, multiply by the weight, and sum for a total. Write your weights down and share them with the supplier. A shop that knows you weight OTD at 30% will behave differently from one that thinks all seven metrics count equally. Transparency about weighting is not giving away leverage — it is telling your supplier exactly how to keep your business.

Sensible starting weights, to be adjusted: OTD 25%, first-pass yield 20%, NCR handling 15%, documentation 15%, quote speed/accuracy 10%, communication 10%, cost behavior 5%. A medical buyer might shift documentation to 25% and NCR to 20%. A production buyer might push OTD to 35%.

A worked scorecard

Below is a complete monthly scorecard for a single supplier, using the production-buyer weighting above. Each metric is scored 1–5 against defined bands, multiplied by its weight, and summed. This is the artifact you copy into a spreadsheet and fill in every month.

A total of 4.15 out of 5 (83%) is a solid, keep-and-grow supplier with two clear improvement areas: documentation (they are delivering, but you are still occasionally asking) and NCR closure speed. Notice how the weighting steers attention — the documentation gap costs 0.45 of a possible 0.75, so it drags the total more than the perfect quote-speed score lifts it. That is the scorecard doing its job: pointing you at the conversation worth having.

Set thresholds for action: above 90% is a preferred supplier you give more business; 75–90% is stable with a documented improvement plan; below 75% triggers a formal review and, if the trend does not turn, dual-sourcing. Publish these thresholds too.

How to build and run the scorecard, step by step

1. Pick your metrics and weights first. Start from the seven above, drop any that do not apply to your parts, and set weights that sum to 100% based on what actually hurts you when it fails. Do this before you look at any supplier data so the design is honest.
2. Define the bands in writing. For each metric, write the 5/3/1 thresholds. Ambiguous bands get scored by mood; explicit bands get scored the same way every month by whoever holds the pen.
3. Find the data source for each metric. OTD comes from your ERP or receiving log against PO acknowledgement dates. FPY and NCR come from incoming inspection. Quote metrics come from your RFQ email trail. Documentation comes from a simple checkbox at receiving. If a metric has no data source, you cannot score it — fix the data collection or drop the metric.
4. Baseline three months before you judge. One month is noise. Collect three months, share the picture with the supplier, and agree it reflects reality before anyone's business decision rides on it.
5. Score monthly, review quarterly. Fill the card every month so trends are visible. Hold a quarterly business review (QBR) with the supplier to walk the trend together — more on this below.
6. Close the loop. Every QBR should end with at most two or three agreed actions, an owner on each side, and a date. A scorecard with no follow-up actions is a chart nobody reads.

Notice this whole process depends on the supplier being able to hand you clean evidence. That is where inspection and documentation practice matters — our quality assurance approach exists precisely so the CMM data, FAI reports and material certificates that feed a scorecard are produced as a matter of routine, not scrambled together when a customer asks.

What evidence to ask any supplier for

A scorecard is only as good as the evidence behind it, and you are entitled to ask for that evidence from any shop you buy from. Frame it as a standing expectation, not a special request. For each shipment you should be able to get, without a fight: the FAI report on new or revised parts, CMM inspection data on the critical dimensions you flagged, material certificates traceable to the batch, and an on-time record you can reconcile against your own receiving log.

To be concrete about what "available" looks like: at Sendot we inspect with CMM, issue FAI reports, provide material certificates, and hold ISO 9001. That is offered here as a checklist of what to demand from any supplier, not as a claim that we are the only shop that does it. One honest caveat on certification: ISO 9001 is the certification Sendot holds. If your program requires ISO 13485 for medical or IATF 16949 for automotive, a supplier can often supply much of the documentation those supply chains require — but certification status itself must be confirmed directly, never assumed from a website. Treat any supplier claiming a certification they cannot produce a current certificate for as a red flag on your documentation metric.

Common mistakes that make a scorecard lie to you

Measuring against the wishful date, not the confirmed date. If you score OTD against the date you wanted rather than the date the shop acknowledged, you will punish a supplier for your own optimistic planning and lose the ability to have a real conversation.

Punishing transparency. If a shop that self-reports an NCR scores worse than a shop that ships defects quietly, you have built an incentive for silence. Score the handling of problems, not just their existence.

Blending first articles into production yield. First articles are supposed to catch problems; a low first-article yield is the system working. Segment them out or you will chase a number that means nothing.

Judging responsiveness on your clock. An overseas shop on UTC+8 is asleep when you send your afternoon email. Measure response within their business day, or you are scoring the time zone, not the supplier.

Turning the QBR into a tribunal. A scorecard used only as a stick teaches suppliers to game the metrics and hide the truth. The score is a conversation starter — covered next.

Over-engineering the card. Twenty metrics that nobody has time to populate is worse than seven you actually fill in every month. Discipline beats completeness.

Using a low score as a conversation, not a punishment

The point of a quarterly business review is not to read the supplier their failing grade. It is to sit down with the trend and ask why. A drop in OTD might trace to a bottleneck machine, a raw-material lead-time problem, or your own late CAD releases pushing their whole schedule right. A shop that trusts you will tell you which; a shop that fears you will make an excuse. Which one you get depends on how you have used past scores.

Structure the QBR simply: walk the last quarter's trend on each weighted metric, celebrate what improved, pick the one or two metrics dragging the total, and ask the supplier what they need from you to move them. Sometimes the answer is on your side — cleaner drawings, earlier forecasts, faster approval of first articles. Agree two or three actions with owners and dates, and open the next QBR by checking them. A supplier that sees you act on your own commitments will invest in yours. This is the difference between a scorecard that improves a relationship and one that quietly ends it while both sides pretend otherwise.

Frequently asked questions

A scorecard is worth building only when you have a supplier worth measuring. If you are still choosing, start with the one-time overseas CNC supplier vetting checklist, then bring the winner onto the scorecard in this article. And if you want a shop whose CMM reports, FAI documents and on-time record are built to feed a scorecard from day one, see our CNC machining services or send a drawing through request a quote — you will have a number back within 12 hours to start scoring against.