Since you seem unable (likely understandably so) to give any more information, I am going to rephrase your situation so you can determine if we are on the same level. And make an assumption that looks like reasonable.
For all intent and purposes the symmetric keys are not keys but just data, since we get them "prefabricated" out of some system. Thus, in other words, you are asking how can I send some "private data" (= keys) to a group of "users" (= devices) over an "insecure channel" (= the factory and it's employees), with the users having an uniquely identifying feature (= serial number) which is easily known and/or imitated by a malicious user.
I will also amuse losing some devices (or having a cost of reprogramming) due to an attack is not a problem.
What you need is for each user to be able to request its data only ONCE.
- The user connects to your server using a secure authenticated channel (I would suggest some kind of SSL connection).
- The user sends its ID (serial number) to the server.
- The server checks if the data (symmetric key) corresponding to this ID has not been handed out (is not flagged).
- Is not flagged: The server sends the user its data (symmetric key) and flags this key as handed out in it's DB-system.
- Is flagged: The server does not deliver the data to the user.
The device should only be able to work if it has received a symmetric key from this server. This could be a final part of the production process, let the device initialize (get the key) and check that it has received a key (without the device letting know that the key is of course).
Should someone breach your factory and start sending malicious requests to the server, it cannot breach the security (though it will involve some costs due to some devices being manufactured with already used serial numbers, either the cost of a device must be very small or you can reprogram the serial number, which should also reset the symmetric key of course).
The whole point of this system is to reduce the size of the secure environment, only the server and its DB need to be secure and those can be located off-site if needed. Assuming the server and DB are safe and the device cannot be read or abused between initialization and being in the customers possession, once a device leaves the factory, the fact it turns on will let the customer know it has its key and only the server (and DB) know this key.
I believe the device cannot be read or abused between initialization and being in the customers possession falls outside the context of this SE as it is about physical security. One note I do like to make is that if the device could be initialized in front of the customer, say by a vendor under customer supervision this could greatly decrease the potential for this condition to impact security.
Edit: response to remarks made by Gilles
the server has no way to know whether the serial number sent in the request is genuine
I read the comment from the OP saying that "You can assume for this question that keys have already been generated and have an associated identifier" as "there is a DB containing {serial-number, key} pairs". Thus the server can check (does) if the serial number is genuine.
it means that some devices will be bricked as communication breaks down for one reason or another
I'm not sure how to interpret this but I read this as "if communication breaks down during initialization then the device becomes bricked".
I presumed this would be a network internal to the factory or at least the same company making communication problems rather unlikely (though possible, true). Only when the communication fails during sending of the key to the device is there a problem by the way.
When I said "This could be a final part of the production process" I did not mean this should happen outside the companies network,
just that it's the most secure to have it as close to the end as possible, sorry for any miscommunication.
it enables a trivial DoS attack where an attacker claims all serial numbers
It does allow a DoS attack against the serial numbers. This is because I made the assumption that security was a very important concern for this device and that resetting a device would be an acceptable trade-off to security.
However as you will need to be breaching the companies internal network first I do not see this to be that big of a risk. However the attacker will still need to guess correct serial numbers and given a decent lengthy serial the hit-rate should be rather low as we know the production capacity and thus can limit the query limits of the server as an extra security measure.
It also allows quite a wide range of impersonation scenarios that the “only once” rule does not prevent, for example: a company orders 100 devices, provisions 90 and keeps the last 10 as backups; an attacker impersonates one of those 10 and gets a key that the company has recorded as belonging to one of its legitimate devices. You need to provision a key (not necessarily the one that will be used by the final customer, it can be a temporary provisioning key) before the device leaves the manufacturer's secure supply chain.
I should have made it clear that the "initializing" should take place in the companies secure network. Either as part of the production process or as part of the sales (I assumed part of the company but did not make this clear in my "note") but I did say in my "note" that the vendor did have to initialize the device (under supervision of the customer). So any scenario where a customer has a not initialized device should not occur.
I don't think this question is answerable, given that it is extremely general and does not say anything about the device manufacturing process, the distribution process, or the device capabilities.
I do agree that given the general nature of this question an optimal (or very good) solution can not be given. The solution I provide has an inherent cost in the possibility of needing to reprogram serial numbers (or bricked devices depending on implementation). But this is intended to allow a general answer without impacting security. I believe a better solution would involve PKI for client authentication but this is prohibited by the OP.