sarproz-group

Big dataset processing

Viewing 5 reply threads
  • Author
    Posts
    • #4942
      bridgetwang
      Participant

        Hi,

        I’d like to ask when I have a big dataset of 280 images, and want to process it in several subsets, can I coregister all 280 images one time, and then just select less images to form differenct subsets?

        Is it correct that just keep the images I want in ‘DataSet.txt’ to get the subset combination? I’ve tried by deleting half of the dataset in ‘DataSet.txt’, but then I get an error :”SWITCH expression must be a scalar or string constant” What does it mean?

        Another question is similar, when I import some images, and just want to keep the image pairs within e.g. 600m baselines, Is it correct that just delete the images exceeding 600m baselines in ‘DataSet.txt’ ? Because Everytime I set a threshold for baselines, some images will not be connected to the master, and matlab will warn that there are images without connections. I’m not sure should I just ingore the warning or deleting those imgs in ‘DataSet.txt’ or in the SLC folder?

        Best regards

      • #4943
        bridgetwang
        Participant

          Please find the attached error files, thanks a lot !

          Attachments:
          You must be logged in to view attached files.
        • #4946
          periz
          Keymaster

            when I have a big dataset of 280 images, and want to process it in several subsets, can I coregister all 280 images one time, and then just select less images to form differenct subsets?

            you can, it’s your option. Pros: all subsets will share the same master, so, pixels are directly comparable, you can merge/analyze results and have the full historical series. Cons: if your area is affected by temporal decorrelation/changes, keeping long temporal baselines will decrease the overall coherence.

            Is it correct that just keep the images I want in ‘DataSet.txt’ to get the subset combination? I’ve tried by deleting half of the dataset in ‘DataSet.txt’, but then I get an error :”SWITCH expression must be a scalar or string constant” What does it mean?

            yes, it is a possible way. we have no idea about the error you are getting because we do not know what you are doing. if you remove images from dataset.txt, just load your project, the software will only consider the images listed in dataset.txt. the error you get comes from SLC data processing. you do not need that module, what are you doing??

            Another question is similar, when I import some images, and just want to keep the image pairs within e.g. 600m baselines, Is it correct that just delete the images exceeding 600m baselines in ‘DataSet.txt’ ?

            no. the dataset selection module allows you to select images/image pairs to keep/exclude. you do not need to modify the dataset.txt. you can modify the dataset.txt but that’s just for removing images. if you want to keep/discard image connections, if you want to adopt different images graphs, if you want to use geometric baselines for that, it’s easier to do this via the dataset selcetion module (images/connections selection)

            best

            • #4947
              bridgetwang
              Participant

                Thank you very much for your rapid and detailed answers!

                1.

                Cons: if your area is affected by temporal decorrelation/changes, keeping long temporal baselines will decrease the overall coherence.

                For the cons you mentioned, I wonder if I individually corigester let’s say 50 images ; On the other hand, I select the same 50 images from the 280 coregisterd imgs. So the 1st and 2nd 50 images are the same, just they are coregistered to different master. And after same PSI processing, is there going to be much different result between the 1st 50 imgs and 2nd 50 imgs?

                I’d like to divide the 280 images by 2 years, meaning the temporal baseline of every subdataset is within 2 years. I suppose in this case the long temporal baselines problem won’t exist, right?

                2.
                As for the error I come across, I got this error after I delete some imgs in Dataset.txt. When I load the prohect again, I got this error. I’m sure I didn’t change the SLC folder, and already complete the coregistration. How can I fix it ?

                3.
                According to your answer, in order to keep some image connections and discard those exceeding 600m baseline threshold, I should just set the theshold via the dataset selcetion module to get the connections I want, and ignore the matlab warning? In that case, when I come to APS, it will be e.g. “Images Nr. 58 Connection Nr. 39”, is this OK for next processing ?

                Best regards

            • #5951
              bridgetwang
              Participant

                Dear Prof. Daniele,

                I had asked in the forum that I coregistered and processed a long time series dataset including 280 images, and then I changed the “Dataset.txt” to process different subsets individually in one project. Here is the question link:Big dataset processing

                The 3 subset is 2010-2012,2012-2014,2014-2016, and the whole dataset is from 2010-2016. I use the same threshold of APS(0.75) and MISP(0.7) for the whole dataset as well as for different subsets. After I process in this way, I find the number of PS points is not what I expected. The PS points I get in APS and in MISP for subsets are excatly the same number as I get from the whole dataset: for the whole 280 images from 2010-2016, I get 30694 PSC in APS and 334472 PS in MISP. The number I get is exactly the same for subset 2010-2012, subset 2012-2014. subset 2014-2016.

                This is not what I expected. As the urban area experience fast building construction, the extracted PS points of subset should be more than the that of the whole dataset which is affected by temporal decorrelation/changes.

                It is true because I then build a new project, choose the same area and coregister the 2010-2012 dataset and process PSI, I do get more PS with the same threshold.

                I could get more PS points in an individual project as I expected, while not in the whole dataset project. What should I do except from changing the Dataset.txt to extract more PS points for different subset just like in an individual project?

              • #5953
                periz
                Keymaster

                  if you select your points based e.g. on the amplitude stability index == ASI > 0.75 and you get the same amount of points in subsets A, B and C, the explanation is only one: ASI is the same in the 3 datasets.
                  Instead, for each subset you have to re-process the ASI so that you will get 3 quantities: ASI_A, ASI_B and ASI_C
                  best

                  • #5955
                    bridgetwang
                    Participant

                      Thank you so much for the prompt reply. Now I understand where the problem is.

                      So does it also mean that, everytime if I change the graph e.g. from STAR to Full graph or MST, I should re-process the ASI for the current graph?

                  • #5956
                    periz
                    Keymaster

                      no.
                      ASI and reflectivity maps are calculated using the set of images of your dataset, independently from the images graph (which is something related to operations involving pairs of images)
                      this means, if you change the set of images, you have to reprocess ASI and REFL maps

                      • #5961
                        bridgetwang
                        Participant

                          Now it is totally clear to me. I appreciated all of your answers, thank you!

                    Viewing 5 reply threads
                    • You must be logged in to reply to this topic.