Cleaning up a file mess?

[MJuric]

My company switched over to Solidworks in the 2005-2006 time frame. Prior to that the majority of files where 2D DWG files. For some time there was some dabbling with Inventor but most of those files were exported to 2D DWG as well.

The problem I have is that we have several product lines that have been "Migrated" to SW and into the PDM over that time to present. Migrated is in quotes because, well, it didn't ever really happen.

For instance I'd say 90% of the assemblies that are released and are on the PDM have several components that are NOT on the PDM. These most likely were redrawn in SW but never moved to the PDM.

The problems I have are.

• 1) We are making component parts from files that outside the vault, IE not revision controlled...which is not our standard and certainly not optimal
  2) We have assemblies where some parts are in the vault and other parts are not...insanity
  3) We have duplicates, even triplicates and worse of the same file name because files were copied, saved as, etc etc rather than moved and or deleting the original
  4) We have no way, short of opening the drawing and comparing, to know which of these duplicates are the actual latest drawing.

In the beginning someone set this up correctly but clearly there was never any follow thru. We have a "Bulk Check in" folder on the PDM. If you move a drawing to this directory and check it in it automatically populates the data card with the correct Rev level pulled from the drawing. Theoretically we should have moved entire families of parts and all their components, checked them in, moved them to the release folders and released them. That would have been rather easy....but alas someone needed somethnig "Right away" and 15 years later we have a complete unadultered mess on our hands.

So short of the obvious and painful method of essentially moving all these files one by one and dealing with all the broken links etc one file at a time, does anyone have any super cool tricks for dealing with something like this? Even if it only does part of the work...anything would be helpful at this point.

[JSculley]

So short of the obvious and painful method of essentially moving all these files one by one and dealing with all the broken links etc one file at a time, does anyone have any super cool tricks for dealing with something like this? Even if it only does part of the work...anything would be helpful at this point.

If you have an assembly that you believe to be a 'correct' one, you can move it into the vault. Just the assembly, not the parts. Then use Tools....Update References which will give you a dialog showing all the references. From there you can direct PDM to search the entire vault for any missing references (by looking for matching file names) or it can copy any files from outside the vault in to the vault. You can also replace any references with others.

I used this tool a lot when moving our 18000 parts numbers (parts, assemblies, drawings) in to the vault.

[MJuric]

If you have an assembly that you believe to be a 'correct' one, you can move it into the vault. Just the assembly, not the parts. Then use Tools....Update References which will give you a dialog showing all the references. From there you can direct PDM to search the entire vault for any missing references (by looking for matching file names) or it can copy any files from outside the vault in to the vault. You can also replace any references with others.

I used this tool a lot when moving our 18000 parts numbers (parts, assemblies, drawings) in to the vault.

Nice, that will be very helpful.

[MJuric]

If you have an assembly that you believe to be a 'correct' one, you can move it into the vault. Just the assembly, not the parts. Then use Tools....Update References which will give you a dialog showing all the references. From there you can direct PDM to search the entire vault for any missing references (by looking for matching file names) or it can copy any files from outside the vault in to the vault. You can also replace any references with others.

I used this tool a lot when moving our 18000 parts numbers (parts, assemblies, drawings) in to the vault.

Is there a reason none of the "Outside the vault" files show up on the reference list? I looked around and did some searching but nothing mentions anything about this.

I'm wondering if it has to do with the state the file is in. If I run this on a released file it shows nothing. If I change the state to "Clarification" then it shows all the files in the PDM. I'm wondering if I will need to change this to an Engineering change state for this to work.

[jcapriotti]

I'm wondering if it has to do with the state the file is in. If I run this on a released file it shows nothing. If I change the state to "Clarification" then it shows all the files in the PDM. I'm wondering if I will need to change this to an Engineering change state for this to work.

I think that tool uses the database, and those files outside PDM were never recorded by PDM since they weren't in your database. We have the setting on check-in to not allow check-in if any references are outside the vault.

You might try creating another PDM bulk folder, something like, "To migrate". Add all remaining files and check them in. Then duplicates/triplicates" will fail giving you or someone the task of cleaning that up. Next, check the SolidWorks setting "Reference Documents specified in File Locations" and set a "Referenced Documents" path to this "To Migrate" folder. Next you need to checkout all of the assemblies in PDM in check them back in and hopefully it should pickup these "To migrate" files. You may have to un-map or rename the network files location so the assembly can't see them. The check-in setting for outside files should prevent the check-in if it still loads the outside file.

I'm sure I'm missing a step as I'm going on memory here and don't have PDM in front me at the moment.

[MJuric]

I think that tool uses the database, and those files outside PDM were never recorded by PDM since they weren't in your database. We have the setting on check-in to not allow check-in if any references are outside the vault.

So the only "Out of PDM" files it will find are those that were at one time in the PDM and then removed from the PDM? That seems a bit unhelpful.

You might try creating another PDM bulk folder, something like, "To migrate". Add all remaining files and check them in. Then duplicates/triplicates" will fail giving you or someone the task of cleaning that up. Next, check the SolidWorks setting "Reference Documents specified in File Locations" and set a "Referenced Documents" path to this "To Migrate" folder. Next you need to checkout all of the assemblies in PDM in check them back in and hopefully it should pickup these "To migrate" files. You may have to un-map or rename the network files location so the assembly can't see them. The check-in setting for outside files should prevent the check-in if it still loads the outside file.

I'm sure I'm missing a step as I'm going on memory here and don't have PDM in front me at the moment.

I had thought about this but my big issue is that at the moment all the files "Outside" the vault are actually "Live" IE we are making and selling them. I'd have to check what/Who I would completely hose by placing them someplace "Completely new"

Thanks again

[JSculley]

So the only "Out of PDM" files it will find are those that were at one time in the PDM and then removed from the PDM? That seems a bit unhelpful.

No, that's not how it works. You may need to check the assembly file in, without checking in all the referenced files. This was 12 years ago, so I'm fuzzy on the details but I know for a fact that files that were never in the vault could be brought in.

I'm pretty sure I documented the process, but a preliminary search for the document hasn't turned anything up. It may be on my old workstation. I'll have a look tomorrow.

[MJuric]

No, that's not how it works. You may need to check the assembly file in, without checking in all the referenced files. This was 12 years ago, so I'm fuzzy on the details but I know for a fact that files that were never in the vault could be brought in.

I'm pretty sure I documented the process, but a preliminary search for the document hasn't turned anything up. It may be on my old workstation. I'll have a look tomorrow.

Ok, thanks.

I'll have to play with it a bit more but right now when I open an assembly that is checked into the vault and open the PDM in the Task plane it shows several parts as "Not in the vault" but they do show up in the assembly so they are actually there. When I do the "Update references" in the PDM explorer it only shows the parts that are in the vault. IT shows absolutely nothing unless the state of the part is, for us, "In clarification"

[jcapriotti]

@MJuric Does the contains tab show the outside files? I didn't think it saved that information.

Now SolidWorks 'knows' were it expects to find the files, whether inside or outside PDM. As long as the outside files are there, it will load them. If you move them to the new bulk load PDM folder and put a file reference path there and tell SolidWorks to use that path, it will load it from there and then record the PDM location on check-in.

I understand you have designs that need these files both inside and out. Once in PDM, these files need to become the master. No outside edits are allow and any changes in PDM means someone needs to copy it out and overwrite the outside copy. Only way to keep your sanity intact.

[JSculley]

Ok, thanks.

I'll have to play with it a bit more but right now when I open an assembly that is checked into the vault and open the PDM in the Task plane it shows several parts as "Not in the vault" but they do show up in the assembly so they are actually there. When I do the "Update references" in the PDM explorer it only shows the parts that are in the vault. IT shows absolutely nothing unless the state of the part is, for us, "In clarification"

So, I just did a little test. I created three new parts, put them in a new assembly and saved it all outside the vault. I then copied the assembly and one of the parts in to the vault. Before checking them in, I used Tools....Update References and this is what it looks like:

If you aren't seeing this behavior, something else is going on.

[MJuric]

So, I just did a little test. I created three new parts, put them in a new assembly and saved it all outside the vault. I then copied the assembly and one of the parts in to the vault. Before checking them in, I used Tools....Update References and this is what it looks like:

image.png

If you aren't seeing this behavior, something else is going on.

This is what I see.
When I open a part in SolidWork and look at the PDM tab it looks like this. Bunch of parts not in the PDM, but they do show up in the assy I just opened...well most of them, one appears to be just missing.

When I click on the part and say "show references" without it in the released state I get nothing.

When I change the part to "clarification", (We have "For clarification" and "For change") I see this. None of the parts outside the vault show up.

I've not tried changing the state to "for change" to see if this is an issue with our states or not because that forces a rev level change and that all rolls down hill and causes a bunch of changes downstream that I'd rather not trigger. Technically, for us, "For clarification" should be the same state as "For change". The difference is that we use clarification for items that have no effect on manufacturing of the part. Things like Missing dimensions, added views for clarity....or moving files. "Request change" are changes that do effect manufacturing and this ends up being a Rev level and has to be sent thru a couple departments for approval, ERP changes etc.

[JSculley]

This is what I see.
When I open a part in SolidWork and look at the PDM tab it looks like this. Bunch of parts not in the PDM, but they do show up in the assy I just opened...well most of them, one appears to be just missing.
image.png
When I click on the part and say "show references" without it in the released state I get nothing.

image.png
When I change the part to "clarification", (We have "For clarification" and "For change") I see this. None of the parts outside the vault show up.
image.png

I've not tried changing the state to "for change" to see if this is an issue with our states or not because that forces a rev level change and that all rolls down hill and causes a bunch of changes downstream that I'd rather not trigger. Technically, for us, "For clarification" should be the same state as "For change". The difference is that we use clarification for items that have no effect on manufacturing of the part. Things like Missing dimensions, added views for clarity....or moving files. "Request change" are changes that do effect manufacturing and this ends up being a Rev level and has to be sent thru a couple departments for approval, ERP changes etc.

OK. Another quick test shows me that you need to have check out rights for the assembly. We have a setup similar to yours. Files in the 'Approved' state can't be checked out. Files in the 'Non-revision Change Pending' state (which is like your 'For clarification' state) can be checked out. When logged in as a 'normal' user, if I try to update references on an assembly in the 'Approved' state, I get the blank window like yours. For an assembly in the 'Non-revision Change Pending' state, I can update the references. If I log in as an admin user who can check out files in all states I can update the references.

However, when I *can* update references, they are all shown, even the ones outside the vault, so there is another piece of the puzzle still missing. When you open the assembly in SOLIDWORKS and click on 'File...Find References' what does it say?

[MJuric]

OK. Another quick test shows me that you need to have check out rights for the assembly. We have a setup similar to yours. Files in the 'Approved' state can't be checked out. Files in the 'Non-revision Change Pending' state (which is like your 'For clarification' state) can be checked out. When logged in as a 'normal' user, if I try to update references on an assembly in the 'Approved' state, I get the blank window like yours. For an assembly in the 'Non-revision Change Pending' state, I can update the references. If I log in as an admin user who can check out files in all states I can update the references.

However, when I *can* update references, they are all shown, even the ones outside the vault, so there is another piece of the puzzle still missing. When you open the assembly in SOLIDWORKS and click on 'File...Find References' what does it say?

So I happened to be on a different system because mine is being used for a remote session. I tried to open the same file and got a "Can't find file". Turns out the system I was on did not have the drive that the "Outside of PDM" files are on mapped. So I mapped the drive and open the assy.

I THEN went to the PDM and did "find references" and all the parts showed up.

So my guess here is that I need to somehow break the existing references and then force SW to refind them and then the PDM will find them.

[matt]

So my guess here is that I need to somehow break the existing references and then force SW to refind them and then the PDM will find them.

Use that reference search list to your advantage. The first place it looks is files that are already open. So if you open all the references, regardless of where they are located, SW will use those files (as long as the file names are correct).

You could also collect all the files and put them in the sandbox, or local cache or whatever you call the local folder that PDM puts the files in when it copies them out of the vault.

There are a lot of ways to win the reference management struggle with SW, but the best way is to use the rules to win.

[bnemec]

What a headache this all is. Sounds like how our migration into the vault then migration to new vault went. In my tiny bit of experience, to migrate more than a couple thousand files needs to make use of APIs, even if you have free labor there is still the human factor of inconsistency; computers are great at doing the same thing over and over exactly the same each time.

A couple notions I picked up along the way that seem to really complicate migration of files in PDM.

1) PDM is oblivious to files that are not in that vault. They will not be found in the database, they will not show up in contains or where used tabs there is no record of them in PDM.
2) the life or scope of a file in PDM is different than what we're used to in file systems. In a file system if you delete a file it's gone, it the human creates a new file with the same exact name the human and CAD see it as a new file.
a) In PDM you can check out a file and save a new file (example dumb solid of a physical object being remodeled with full feature tree) can be saved over it then checked back in; as far as PDM is concerned it's still the same file (same Document ID in the PDM Database), but CAD will know that internal IDs are different.
b) If the user (or the CAD program) deletes a file in vault and saves a new one with the same name it's a new file in PDM with a new Document Number. The old file is still in PDM database and archives, it has a different DocumentID but same name as the new file. They will stay there until they are "Destroyed" hopefully only your admins have permissions to destroy files. This may be what @JSculley is talking about that once a file is in vault it can be deleted but PDM still keeps track of it even though it doesn't show up anywhere. They can be found by right clicking in a vault view "white space" -> Properties -> Deleted Items tab.

3) if you have two vault views to different vaults on one PC they are mutually exclusive. Each vault is completely walled off from the other,
4)Solidworks (and some other CADs as well) will "fix" file refs for you by using the wrong file by name, not full path. In Solid Edge I learned how to break it's fingers and cut it's legs off so that I could ALMOST prevent it from "fixing" file ref; I have not yet learned how to do that to Solidworks.
5) Outside of the function of PDM Add in, Solidworks is oblivious that C:\Vault is a special folder. It doesn't care that the path is a network share, local drive or PDM; it seems that CAD treats them all the same.
6)The add in displays a mixture of PDM Database data, local cache and file refs from the parent file. This can be tweaked from the PDM Add in settings.

7) When there are files by same name in and out of the vault, I found it very difficult to see exactly which path Solidworks opened. Solid Edge had a tool that was verbose in telling you which file(full path) it was using; I've found Solidworks Pack and Go to be most efficient for informing me Sometimes unmounting network shares and/or deleting other vault views (remove local cache) is a good help to make sure SW only uses the files you're expecting.

[bnemec]

Use that reference search list to your advantage. The first place it looks is files that are already open. So if you open all the references, regardless of where they are located, SW will use those files (as long as the file names are correct).

You could also collect all the files and put them in the sandbox, or local cache or whatever you call the local folder that PDM puts the files in when it copies them out of the vault.

There are a lot of ways to win the reference management struggle with SW, but the best way is to use the rules to win.

The problem is that reference list may remain the same from client to client, but the contents of those paths will vary and it's almost impossible to predict what the various client machines look like. Well, at least it was/is for me, maybe others have it nailed down. In short, I wish that the Solidworks PDM Add-in would grab hold of how Solidworks looks for referenced files so that the first is memory, second is PDM (not local cache, but actual PDM search)

One example is if one parent file was checked into the vault too soon and PDM didn't resolve reference file. Later that referenced file is added to the vault. From then on other parent files will resolve that reference when they are checked in. But that first parent will not be updated, so PDM doesn't know it needs a file that is now in the vault. When you ask pdm to update refs it reads the refs in the SW file that still point outside the vault (SW parent files store the full path that was referenced) so PDM says that full path is outside the vault. As it is now a file that SW is looking for may be in the vault, but if it is not resolved in PDM as a reference to the parent that SW is opening it's a crap shoot if the client has the referenced file cached or not. PDM will only cache the referenced files it has resolved, regardless of what SW will need. And SW doesn't know that there's a difference between what's in local cache and what all is actually in the vault so it's going to use that list @matt mentions on open. So if you can master that list, then maybe SW will update to use the file that's in vault instead of the one on the network share. That brings me back to my original point, you must get that list and behavior the same on all of the clients; which in my experience is nearly impossible if you have more than just a couple users.

Maybe I'm just muddying the waters. can @Diaval shed some light on this?

[MJuric]

What a headache this all is.

Thanks for the information. I think the most important step would be "Don't piece meal it". If you're going to migrate data, migrate ALL the data. Would have been 100 times easier if that would have been done. Right now it's essentially a "File by File" situation and it's a real PIA.

[bnemec]

Thanks for the information. I think the most important step would be "Don't piece meal it". If you're going to migrate data, migrate ALL the data. Would have been 100 times easier if that would have been done. Right now it's essentially a "File by File" situation and it's a real PIA.

Cannot stress that point enough.

It may be special to our dataset, but I learned the slow agonizing way that you cannot expect users that have never seen PDM to be able to migrate "the stragglers" into the vault. There's so many factors at play in both SW and PDM and having a complete understanding of all those things are really out of scope for the average Designers. I know some will argue this point and say that anyone using CAD needs to understand how to use the tool. Not really true in every case. For those who've made a living of CAD itself will see it from one perspective, but some forget that CAD is just one of the many tools in the box. I'm digressing.

My point is, new PDM users cannot be trusted to migrate data into a vault when the entire data set is a tangled web of references. If each project has it's own top level and all the sub assemblies and part files are unique to that top level then they'll probably be fine. I learned a lot about our dataset when I indexed all of our Solid Edge files; to sum it up if you give me one solid edge file path I could touch every other solid edge file in our data set just using recursion up and down the reference tree, there would be no file on the old network shares that was an island. So if you have files with a lot of where used then keeping refs correct while migrating requires a clear, consistent plan. Coincidentally, if you can actually come up with a clear and consistent plan you've done over half of the work of writing a program to do the work.

[HerrTick]

About the best you can do is dump a few gallons of formaldehyde on the festering mess and move forward. Preserve the carcass for posterity, and take what you need as you need it.