[olug] Grep Sed Question
Stan Coleman
stan at srcproductions.com
Mon Jun 18 17:30:42 UTC 2007
Ops ment to send this to the group.
----- Forwarded message from stan at srcproductions.com -----
Date: Mon, 18 Jun 2007 12:26:45 -0500
From: Stan Coleman <stan at srcproductions.com>
Reply-To: Stan Coleman <stan at srcproductions.com>
Subject: Re: [olug] Grep Sed Question
To: Will Langford
I guess I should have provided a little more input. Didn't think
anybody would write a nice perl script for me (learning perl is on my
list of things to do) so I was trying to do the project with my
limited knowledge of grep, awk and sed. Perl I'm sure would be easier
and much better than a bash script.
Here's what I'm trying to do. ANY suggestions are welcome. Lets say
we have a device I'll call Bulliten Board that came with software to
do updates with. Lets call that software Provided Software. Lets say
the Bulliten Board also has the ability to upload data to it via FTP
as well as the Provided Software. All pages are controlled by a text
file on the Bulliten Board called playlist.txt. What I want to do is
upload pages from one FTP server (lets call that FTP server External
FTP) to the Bulliten Board even X minutes . Actually I already have
that script up and running. What I want to do now is to make sure that
the pages from the External FTP server stay updated correctly even if
somebody logs onto the Bulliten Board with the Provided Software.
Since the Provided Software doesn't know what I'm uploading via FTP
from my External FTP server the Provided Software would remove any
lines added or changed by External FTP server. I need a way to put the
External FTP pages (lines in playlist.txt) back on if the Provided
Software removes or modifies them in the playlist. If I could get the
provided software and the external FTP server to talk to each other I
wouldn't have a problem but to make a long network story short
politics would prevent such a thing.
I'm sure this is as clear as mud by now. Since the provided
software doesn't know what I'm uploading via my external FTP server to
the FTP server on the Bulliten Board I just thought I'd pole the
Bulliten Board every X minutes and simple rebuild anything that the
Provided Software might be removing or changing. The Provided Software
only updates manually every couple of days (or weeks) so it's not a
huge problem. Since I'm poling the box every X minutes already I just
figured I'd take the playlist apart and rebuild it with the data
sitting on my external FTP server.
That's were the grep sed question came in. My thinking was to check
for any pages the external FTP Server had uploaded at the start of my
script and then simply remove them. Then at the end of the script I'd
cat the changes back in. That way if somebody makes a change on the
Exteranl FTP server it gets changes every X minutes when I do other
tasks already. Then if there are no changes on the external FTP Server
it simply puts the original lines back in. [The more I write to help
clear this up the more confused your probably getting.]
I'm sure somebody that's done a lot of Perl scripting is say, "Why
would you want to go to all that trouble?" The answer is you work with
what you know until you learn something else.
Example of a playlist.txt file:
2.00 *632803561124310484
, , , ,000000, , ,<DISSOLVE> S
SU, , , ,000000, , ,Food-Monday
, , , ,000000, , ,<DISSOLVE> S
M, , 000000,130000,000000, , ,Food-Monday
, , , ,000000, , ,<DISSOLVE> S
M, , 130000, ,000000, , ,Food-Tuesday
, , , ,000000, , ,<DISSOLVE> S
T, , 010000,130000,000000, , ,Food-Tuesday
, , , ,000000, , ,<DISSOLVE> S
T, , 130000, ,000000, ,
,Food-Wednesday
, , , ,000000, , ,<DISSOLVE> S
W, , 000000,130000,000000, , ,Food-W
, , , ,000000, , ,<DISSOLVE> S
06182007,07042007, , ,000010, , ,July4th-1
, , , ,000000, , ,<DISSOLVE> S
MW, , 080000,110000,000008, ,
,Whatchamacalit
File is divided into two lines for each page. Information on first
line is just
for an effect (but must be there). Second line contains the start
date, end date,
start time and end time. Page name is at the end of the second line.
While it would be a piece of cake to grep for say, "Whatchamacalit"
I still need
to get the line before it removed as well, that is unless somebody
comes up with
a better way to do it.
Another thought I had was to just reserve the last 10 lines or so for
the external
FTP Server lines and the use sed to remove the last 10 lines of the
file before I
rebuild the list. Of course that means the I have to do more education
for the person
running the Provided Software.
[In case somebody is wondering I'm not making money off this
project. It's for a public
school]
>
> On 6/16/07, Christopher Cashell <topher-olug at zyp.org> wrote:
>>
>> At Fri, 15 Jun 07, Unidentified Flying Banana Stan Coleman, said:
>>
>> > I have a project where I want to remove a specific line from a text
>> > file. Piece of cake I've done that many times. Here is the twist I
>> > want to remove the specific line AND the line just prior to the
>> > specific line. Any thoughts?
>>
>> I think that because of the line oriented nature of sed and grep, this
>> will be difficult, if not impossible to accomplish with them. By the
>> time they know that you've reached the match line, they've already
>> printed out the preceding line.
>>
>> My suggestion would be to use awk or perl, provided there isn't some
>> special restriction or requirement for grep or sed.
>>
>> I threw together a quick example in perl, which I've included here.
>> It's only been tested against one little sample file I threw together,
>> but it seems to work. Note that you will have to change /tag/ to
>> whatever regular expression will match the line you want removed.
>>
>> It's currently setup to act as a filter, such as:
>>
>> rem_line+pre.pl <input_file >output_file
>>
>> Also note that I haven't written a whole lot of perl lately, so bugs are
>> entirely likely. If anyone happens to see an error, please let me know.
>>
>> --
>> | Christopher
>> +------------------------------------------------+
>> | Here I stand. I can do no other. |
>> +------------------------------------------------+
>>
>>
>> ------------------CUT HERE------------------------
>>
>> #!/usr/bin/perl
>> # vim: ts=3 sw=3 et sm ai smd sc bg=dark
>>
>> use strict;
>> use warnings;
>>
>> my $previous;
>> my $skip;
>> my $match = 'tag'; # Regex to match line to remove
>> $_ = ""; # We need to pre-initialize this so our until loop works
>>
>> # We need to grab the first line here and set $previous to it since
>> # we're buffering 1 line to give us the ability to skip the line
>> # preceding our match.
>> #$_ = <STDIN>;
>> until (not /$match/) {
>> $_ = <STDIN>;
>> }
>> $previous = $_;
>>
>> while (<STDIN>) {
>> if (/$match/) {
>> # Don't print anything, and set $skip so we skip the next one.
>> $skip = "yes";
>> }
>> else {
>> unless ($skip) {
>> print $previous;
>> }
>> undef $skip;
>> }
>> $previous = $_;
>> }
>>
>> # We're operating one line behind, so now we need to print the final
>> # line, provided we're not skipping.
>> unless ($skip) { print $previous }
>>
>> _______________________________________________
>> OLUG mailing list
>> OLUG at olug.org
>> http://lists.olug.org/mailman/listinfo/olug[1]
>>
> _______________________________________________
> OLUG mailing list
> OLUG at olug.org
> http://lists.olug.org/mailman/listinfo/olug[2]
>
Links:
------
[1] http://lists.olug.org/mailman/listinfo/olug
[2] http://lists.olug.org/mailman/listinfo/olug
----- End forwarded message -----
-------------- next part --------------
I guess I should have provided a little more input. Didn't think
anybody would write a nice perl script for me (learning perl is on my
list of things to do) so I was trying to do the project with my
limited knowledge of grep, awk and sed. Perl I'm sure would be easier
and much better than a bash script.
Here's what I'm trying to do. ANY suggestions are welcome. Lets say
we have a device I'll call Bulliten Board that came with software to
do updates with. Lets call that software Provided Software. Lets say
the Bulliten Board also has the ability to upload data to it via FTP
as well as the Provided Software. All pages are controlled by a text
file on the Bulliten Board called playlist.txt. What I want to do is
upload pages from one FTP server (lets call that FTP server External
FTP) to the Bulliten Board even X minutes . Actually I already have
that script up and running. What I want to do now is to make sure that
the pages from the External FTP server stay updated correctly even if
somebody logs onto the Bulliten Board with the Provided Software.
Since the Provided Software doesn't know what I'm uploading via FTP
from my External FTP server the Provided Software would remove any
lines added or changed by External FTP server. I need a way to put the
External FTP pages (lines in playlist.txt) back on if the Provided
Software removes or modifies them in the playlist. If I could get the
provided software and the external FTP server to talk to each other I
wouldn't have a problem but to make a long network story short
politics would prevent such a thing.
I'm sure this is as clear as mud by now. Since the provided
software doesn't know what I'm uploading via my external FTP server to
the FTP server on the Bulliten Board I just thought I'd pole the
Bulliten Board every X minutes and simple rebuild anything that the
Provided Software might be removing or changing. The Provided Software
only updates manually every couple of days (or weeks) so it's not a
huge problem. Since I'm poling the box every X minutes already I just
figured I'd take the playlist apart and rebuild it with the data
sitting on my external FTP server.
That's were the grep sed question came in. My thinking was to check
for any pages the external FTP Server had uploaded at the start of my
script and then simply remove them. Then at the end of the script I'd
cat the changes back in. That way if somebody makes a change on the
Exteranl FTP server it gets changes every X minutes when I do other
tasks already. Then if there are no changes on the external FTP Server
it simply puts the original lines back in. [The more I write to help
clear this up the more confused your probably getting.]
I'm sure somebody that's done a lot of Perl scripting is say, "Why
would you want to go to all that trouble?" The answer is you work with
what you know until you learn something else.
Example of a playlist.txt file:
2.00 *632803561124310484
, , , ,000000, , ,<DISSOLVE> S
SU, , , ,000000, , ,Food-Monday
, , , ,000000, , ,<DISSOLVE> S
M, , 000000,130000,000000, , ,Food-Monday
, , , ,000000, , ,<DISSOLVE> S
M, , 130000, ,000000, , ,Food-Tuesday
, , , ,000000, , ,<DISSOLVE> S
T, , 010000,130000,000000, , ,Food-Tuesday
, , , ,000000, , ,<DISSOLVE> S
T, , 130000, ,000000, , ,Food-Wednesday
, , , ,000000, , ,<DISSOLVE> S
W, , 000000,130000,000000, , ,Food-W
, , , ,000000, , ,<DISSOLVE> S
06182007,07042007, , ,000010, , ,July4th-1
, , , ,000000, , ,<DISSOLVE> S
MW, , 080000,110000,000008, , ,Whatchamacalit
File is divided into two lines for each page. Information on first
line is just
for an effect (but must be there). Second line contains the start
date, end date,
start time and end time. Page name is at the end of the second line.
While it would be a piece of cake to grep for say, "Whatchamacalit"
I still need
to get the line before it removed as well, that is unless somebody
comes up with
a better way to do it.
Another thought I had was to just reserve the last 10 lines or so for
the external
FTP Server lines and the use sed to remove the last 10 lines of the
file before I
rebuild the list. Of course that means the I have to do more education
for the person
running the Provided Software.
[In case somebody is wondering I'm not making money off this
project. It's for a public
school]
>
> On 6/16/07, Christopher Cashell <topher-olug at zyp.org> wrote:
>>
>> At Fri, 15 Jun 07, Unidentified Flying Banana Stan Coleman, said:
>>
>> > I have a project where I want to remove a specific line from a text
>> > file. Piece of cake I've done that many times. Here is the twist I
>> > want to remove the specific line AND the line just prior to the
>> > specific line. Any thoughts?
>>
>> I think that because of the line oriented nature of sed and grep, this
>> will be difficult, if not impossible to accomplish with them. By the
>> time they know that you've reached the match line, they've already
>> printed out the preceding line.
>>
>> My suggestion would be to use awk or perl, provided there isn't some
>> special restriction or requirement for grep or sed.
>>
>> I threw together a quick example in perl, which I've included here.
>> It's only been tested against one little sample file I threw together,
>> but it seems to work. Note that you will have to change /tag/ to
>> whatever regular expression will match the line you want removed.
>>
>> It's currently setup to act as a filter, such as:
>>
>> rem_line+pre.pl <input_file >output_file
>>
>> Also note that I haven't written a whole lot of perl lately, so bugs are
>> entirely likely. If anyone happens to see an error, please let me know.
>>
>> --
>> | Christopher
>> +------------------------------------------------+
>> | Here I stand. I can do no other. |
>> +------------------------------------------------+
>>
>>
>> ------------------CUT HERE------------------------
>>
>> #!/usr/bin/perl
>> # vim: ts=3 sw=3 et sm ai smd sc bg=dark
>>
>> use strict;
>> use warnings;
>>
>> my $previous;
>> my $skip;
>> my $match = 'tag'; # Regex to match line to remove
>> $_ = ""; # We need to pre-initialize this so our until loop works
>>
>> # We need to grab the first line here and set $previous to it since
>> # we're buffering 1 line to give us the ability to skip the line
>> # preceding our match.
>> #$_ = <STDIN>;
>> until (not /$match/) {
>> $_ = <STDIN>;
>> }
>> $previous = $_;
>>
>> while (<STDIN>) {
>> if (/$match/) {
>> # Don't print anything, and set $skip so we skip the next one.
>> $skip = "yes";
>> }
>> else {
>> unless ($skip) {
>> print $previous;
>> }
>> undef $skip;
>> }
>> $previous = $_;
>> }
>>
>> # We're operating one line behind, so now we need to print the final
>> # line, provided we're not skipping.
>> unless ($skip) { print $previous }
>>
>> _______________________________________________
>> OLUG mailing list
>> OLUG at olug.org
>> http://lists.olug.org/mailman/listinfo/olug[1]
>>
> _______________________________________________
> OLUG mailing list
> OLUG at olug.org
> http://lists.olug.org/mailman/listinfo/olug[2]
>
Links:
------
[1] http://lists.olug.org/mailman/listinfo/olug
[2] http://lists.olug.org/mailman/listinfo/olug
More information about the OLUG
mailing list